java.lang.OutOfMemoryError: Java heap space when try to convert Java Object to Json String

Question

I try to convert csv file to Json file 200K of objects where object represents 1 row in csv.

I have Java installed on 32 bit and Project configuration VM arguments: -Xmx1024m

However I get:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuffer.append(Unknown Source)
    at java.io.StringWriter.write(Unknown Source)
    at com.google.gson.stream.JsonWriter.string(JsonWriter.java:478)
    at com.google.gson.stream.JsonWriter.value(JsonWriter.java:328)
    at com.google.gson.Streams.write(Streams.java:113)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Streams.write(Streams.java:124)
    at com.google.gson.Streams.write(Streams.java:136)
    at com.google.gson.Gson.toJson(Gson.java:362)
    at com.google.gson.Gson.toJson(Gson.java:346)
    at com.google.gson.Gson.toJson(Gson.java:260)
    at com.google.gson.Gson.toJson(Gson.java:240)
    at ConvertFromCsv2JsonTWC.init(ConvertFromCsv2JsonTWC.java:186)
    at ConvertFromCsv2JsonTWC.main(ConvertFromCsv2JsonTWC.java:48)

In row:

Gson gson = new Gson();

String output = gson.toJson(container);// <---- crash

for 50k rows it works fine.

This is a template of Json I build:

{
    "crs": {
        "type": "none"
    },
    "type": "FeatureCollection",
    "features": [{
        "geometry": {
            "type": "Point"
        },
        "properties": {
            "ap mac": "00:11:22:33:44:55",
            "ssid": "WiFi",
            "lat": "35.111111",
            "long": "-118.11111",
            "address": "370 xxxxxx",
            "city": "xxxxxxx",
            "state": "CA",
            "zip code": "11111",
            "country": "US",
            "business n": "",
            "location c": "Health Club/Gym",
            "location q": "",
            "indoor fla": "yes"
        },
        "point": [35.390284,
        -118.9929],
        "id": 0,
        "type": "Feature"
    },
          {...},
          ... 
          200000...

So I have 200K objects in properties list

The workaround is to create separate files 20k per each but it not good way.

How can I solve this?

Thanks,

@ElliottFrisch please poste this comment as answer to allow me to accept. Thanks — snaggs
– snaggs, Commented Nov 24, 2013 at 13:50

Elliott Frisch · Accepted Answer · 2013-11-24 14:21:30Z

4

I suggest you use streaming instead of trying to copy it all into a String.

answered Nov 24, 2013 at 14:21

Elliott Frisch

202k20 gold badges166 silver badges265 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Coded9 Over a year ago

How can I stream it if there is only object having different properties and also having a list of 10 million records ??

sethcall · Accepted Answer · 2013-11-24 13:39:42Z

4

You can use the Gson streaming API to instead walk (stream) the data, instead of trying to load it all at once.

answered Nov 24, 2013 at 13:39

sethcall

2,9172 gold badges21 silver badges22 bronze badges

Comments

Egemen Hamutçu · Accepted Answer · 2018-11-08 08:37:48Z

1

This is how you stream data into Json as String using Gson Streaming API

@Nullable
public static String streamContainersIntoJsonString(List<Container> containers) {
    try {
        Gson gson = new Gson();
        ByteArrayOutputStream out = new ByteArrayOutputStream();
        JsonWriter writer = new JsonWriter(new OutputStreamWriter(out, "UTF-8"));
        writer.setIndent("  ");
        writer.beginArray();
        for (Container container : containers) {
            gson.toJson(container, Container.class, writer);
        }
        writer.endArray();
        writer.close();

        return out.toString("UTF-8");
    } catch (IOException e) {
        e.printStackTrace();
    }

    return null;
}

answered Nov 8, 2018 at 8:37

Egemen Hamutçu

1,7523 gold badges22 silver badges39 bronze badges

1 Comment

Attaullah Over a year ago

outofmemory exception for large list

Steve P. · Accepted Answer · 2013-11-24 13:10:53Z

0

There is too much data to read all of it in at once and keep it in memory. You should break it up into smaller parts and process it piece-wise.

answered Nov 24, 2013 at 13:10

Steve P.

14.7k9 gold badges46 silver badges74 bronze badges

Collectives™ on Stack Overflow

java.lang.OutOfMemoryError: Java heap space when try to convert Java Object to Json String

4 Answers 4

1 Comment

Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related