Parse large JSON file with JSON Simple (OutOfMemoryError)

Question

I am trying to parse large JSON file with JSON Simple and i am getting out of memory errors. I am on Windows 10 and my laptop has an 8gb RAM. The file is 250mb, i will also need to parse a 2gb file. I also tried with StrinBuilder but then i am getting memory errors on StringBuilder. Here is my code with StringBuilder:

    BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream("myfile.json")));
    String line = null;
    StringBuilder sb= new StringBuilder("");
    while( (line = br.readLine())!= null ){
        sb.append(line);
    }
    JSONParser parser = new JSONParser();
    Object obj=null;
    try {
        obj = parser.parse(sb.toString());  
    }catch (Exception e) {

    }

and here is the code without StringBuilder:

JSONParser parser = new JSONParser();
        Object obj=null;
        try {
            obj = parser.parse(new FileReader("myfile.json"));  
        }catch (Exception e) {

        }

The error

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.json.simple.parser.Yylex.yylex(Unknown Source) at org.json.simple.parser.JSONParser.nextToken(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source) at org.json.simple.parser.JSONParser.parse(Unknown Source)

Here StringBuilder object holds all the data during iteration. so that all contents in the file is in memory. you can follow the link that show's how to read a big size file. stackoverflow.com/questions/2356137/read-large-files-in-java — Ataur Rahman Munna
– Ataur Rahman Munna, Commented Apr 24, 2016 at 9:28
I suggest you use an event driver pasrer instead of loading the entire JSON as a DOM. This should allow you to process much larger messages. — Peter Lawrey
– Peter Lawrey, Commented Apr 24, 2016 at 9:29
@ataurRahmanMunna i tries what you told and the StringBuilder is fine but i am getting out of memory error on json parser again. — Iraklis Bekiaris
– Iraklis Bekiaris, Commented Apr 24, 2016 at 9:43
In short you can use GSON or Jackson to start with. stackoverflow.com/questions/9390368/… We also have a streaming parser, but it is more designed for low latency. — Peter Lawrey
– Peter Lawrey, Commented Apr 24, 2016 at 9:55
@PeterLawrey I solved the problem with jackson library :). Thanks again Peter. Have a good day — Iraklis Bekiaris
– Iraklis Bekiaris, Commented Apr 24, 2016 at 10:05

Gabriel Silveira · Accepted Answer · 2021-01-16 10:56:40Z

3

If you are open to use other Json parser then you can try Jackson's Streaming API which can be used to parse huge JSON upto even giga bytes of size.It can be used to process huge files without loading them completely in memory.It allows get the data you want and ignore what you don't want also

Read more: https://github.com/FasterXML/jackson-docs/wiki/JacksonStreamingApi

edited Jan 16, 2021 at 10:56

Gabriel Silveira

34 bronze badges

answered Apr 24, 2016 at 10:04

tyagi

2652 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Iraklis Bekiaris Over a year ago

I was able to solve the problem with the help of Peter in the question comments. This is exactly what i used in the end! Thanks anyway!!

Zayn · Accepted Answer · 2019-02-22 00:50:44Z

1

There are some excellent libraries for parsing large JSON files with minimal resources. One is the popular GSON library. It gets at the same effect of parsing the file as both stream and object. It handles each record as it passes, then discards the stream, keeping memory usage low.

Support arbitrarily complex objects (with deep inheritance hierarchies and extensive use of generic types)

Look at this Detailed Tutorial for GSON approach,to solve it problem.

edited Feb 22, 2019 at 0:50

answered Feb 22, 2019 at 0:40

Zayn

5331 gold badge7 silver badges25 bronze badges

Collectives™ on Stack Overflow

Parse large JSON file with JSON Simple (OutOfMemoryError)

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related