OutOfMemoryError while writing JSon array to out put stream

Question

I have Java program and I am reading from file of Size Approx 40GB and converting data on fly and streaming to DataOutputstream.

Below is my code

    JSONObject jsonObj= new JSONObject();
                JSONArray jsonArray = new JSONArray();
                 JSONObject properties = new JSONObject(); 

           while((strLine = in.readLine()) != null)
                {


            if (jsonArray.length()%100 == 0) {

                    printJsonData(jsonArray);
            //      sendJsonData(jsonArray, output);
                    jsonArray = new JSONArray();
                }

if (strLine.trim().contains("data")) {
                        jsonObj.put("properties",properties);
                        jsonArray.put(jsonObj);

    if (strLine.trim().contains("data1")) {
                String secondPart = strLine.split(":",2)[1];
                properties.put("data1", secondPart);    
                continue;
                }
            if (strLine.trim().contains("id")) {
                String secondPart = strLine.split(":",2)[1];
                jsonObj.put("id", secondPart ); 
            continue;

            }

    }

I have two methods one is printing JSON data and other is sending Json data over https.

private void printJsonData(JSONArray jsonArray) {
      int count = jsonArray.length(); // get totalCount of all jsonObjects
      for(int i=0 ; i< count; i++){   // iterate through jsonArray 
            JSONObject jsonObject = jsonArray.getJSONObject(i);  // get jsonObject @ i position 
            System.out.println("jsonObject " + i + ": " + jsonObject);
        }
    }

private void sendJsonData(JSONObject jsonObj,DataOutputStream   output) throws IOException {

    int count = jsonArray.length();     

    for(int i=0 ; i< count; i++) {   // iterate through jsonArray 
        JSONObject jsonObject = jsonArray.getJSONObject(i);  // get jsonObject @ i position 
        System.out.println("Sending Data-->" + jsonObject.toString());
        output.writeBytes(jsonObject.toString()); 
    }
}

Whenver i call print method, every thing works fine. But when i call sendJsonData method. I am getting OutofMemoryerror. Wondering how to fix it?

Where are you getting the value of count in the sendJsonData method? — Michael Markidis
– Michael Markidis, Commented Mar 8, 2017 at 3:00
@MichaelMarkidis int count = jsonArray.length(); I fixed it. thanks — Ammad
– Ammad, Commented Mar 8, 2017 at 3:04
@ScaryWombat I commented the line in while loop for sending data: call is sendJsonData(jsonArray, output); — Ammad
– Ammad, Commented Mar 8, 2017 at 3:05

bichito · Accepted Answer · 2017-03-08 13:56:12Z

1

Since you are not manipulating json in any way I would treat as a text file and send it that way. The likelihood of finding a big JsonObject is good so you may be pushing through too much.

An array of 40gb is awfully big array unless there are huge objects.

** EDIT **

Or you could treat the individual json objects as "files" and push those strings individually

edited Mar 8, 2017 at 13:56

answered Mar 8, 2017 at 13:38

bichito

1,4242 gold badges19 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ammad Over a year ago

Meaning One Object at a time? and No array at all?

bichito Over a year ago

I would try to launch a connection per array element. Treat each element as an string and make an stream of them and send them in a more controlled manner. Maybe you could launch a thread pool to speed things up. Reading the whole thing line by line may be too slow

Ammad Over a year ago

You option for sending One Big Array at a time with one connection help fix this issue. -- Thans Ammad

Collectives™ on Stack Overflow

OutOfMemoryError while writing JSon array to out put stream

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related