"OutOfMemoryError: GC overhead limit exceeded": parse large json file with java

Question

I try to parse large json file (more 600Mo) with Java. My json file look like that:

{
    "0" : {"link_id": "2381317", "overview": "mjklmklmklmklmk", "founded": "2015", "followers": "42", "type": "Gamer", "website": "http://www.google.com",  "name": "troll", "country": "United Kingdom", "sp": "Management Consulting" },
    "1" : {"link_id": "2381316", "overview": "mjklmklmklmklmk", "founded": "2015", "followers": "41", "type": "Gamer", "website": "http://www.google2.com",  "name": "troll2", "country": "United Kingdom", "sp": "Management Consulting" }
    [....]

    "345240" : {"link_id": "2381314", "overview": "mjklmklmklmklmk", "founded": "2015", "followers": "23", "type": "Gamer", "website": "http://www.google2.com",  "name": "troll2", "country": "United Kingdom", "sp": "Management Consulting" }
}

and my code looks like that:

public class dumpExtractor {

    private static final String filePath = "/home/troll/Documents/analyse/lol.json";

    public static void main(String[] args) {

    try {
        // read the json file
        FileReader reader = new FileReader(filePath);
        JSONParser jsonParser = new JSONParser();
        JSONObject jsonObject = (JSONObject) jsonParser.parse(reader);
        Iterator<JSONObject> iterator = jsonObject.values().iterator();

        while (iterator.hasNext()) {
        JSONObject jsonChildObject = iterator.next();
        System.out.println("==========================");
        String name = (String) jsonChildObject.get("name");
        System.out.println("Industry name: " + name);

        String type = (String) jsonChildObject.get("type");
        if (type != null && !type.isEmpty()) {
            System.out.println("type: " + type);
        }

        String sp = (String) jsonChildObject.get("sp");
        if (sp != null && !sp.isEmpty()) {
            System.out.println("sp: " + sp);
        }
        System.out.println("==========================");
        }
        System.out.println("done ! ");
    } catch (IOException ex) {
        ex.printStackTrace();
    } 
    }
}

I 've got this error:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.util.HashMap.createEntry(HashMap.java:897)
    at java.util.HashMap.addEntry(HashMap.java:884)
    at java.util.HashMap.put(HashMap.java:505)
    at org.json.simple.parser.JSONParser.parse(Unknown Source)
    at org.json.simple.parser.JSONParser.parse(Unknown Source)

How I can fix that ?

Thanks in advance.

The problem is the complete object is too huge. Try reading that file line by line and parsing each of the nested objects separately. — Dušan Rychnovský
– Dušan Rychnovský, Commented Sep 11, 2015 at 14:43
Can you add the lines with your import statements so we can see what Parser you're using? — Carlos Bribiescas
– Carlos Bribiescas, Commented Sep 11, 2015 at 14:45
@CarlosBribiescas The stack trace shows it: org.json.simple.parser.JSONParser — Andreas
– Andreas, Commented Sep 11, 2015 at 14:46
docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/… — JimmyB
– JimmyB, Commented Sep 11, 2015 at 14:49
How much RAM do you allow the JVM to allocate? See stackoverflow.com/questions/14763079/… — JimmyB
– JimmyB, Commented Sep 11, 2015 at 14:49

Davide Lorenzo MARINO · Accepted Answer · 2015-09-11 14:52:34Z

3

If you have to read huge JSON Files you can't mantain in memory all informations. Extending memory can be a solution for a file of 1 Gb. If the files tomorrow is a 2 Gb Files?

The right approach to this problem is to parse the json element by element using a streaming parser. Basically instead of loading the whole json in memory and creating a whole big object representing it you need to read single elements of the json and converting them to objects step by step.

Here you find a nice article explaing how to do it with jackson library.

answered Sep 11, 2015 at 14:52

Davide Lorenzo MARINO

27k4 gold badges45 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Andreas · Accepted Answer · 2015-09-11 14:50:04Z

2

You have two choices:

Give more memory to the Java program by specifying the -Xmx argument, e.g. -Xmx1g to give it 1 Gb of memory.
Use a "streaming" JSON parser. This will scale to infinitely large JSON files.

json-simple has a streaming API. See https://code.google.com/p/json-simple/wiki/DecodingExamples#Example_5_-_Stoppable_SAX-like_content_handler

There are other libraries with good streaming parser, e.g. Jackson.

answered Sep 11, 2015 at 14:50

Andreas

160k13 gold badges164 silver badges262 bronze badges

Comments

Rahman · Accepted Answer · 2015-09-11 14:52:45Z

1

Increase the JVM heap space by setting the environment variables :

SET _JAVA_OPTIONS = -Xms512m -Xmx1024m

But it cant be a permanent solution as your file can be increased in future

answered Sep 11, 2015 at 14:52

Rahman

3,7954 gold badges29 silver badges47 bronze badges

Collectives™ on Stack Overflow

"OutOfMemoryError: GC overhead limit exceeded": parse large json file with java

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related