android java not enough sequential memory for stringbuilder

Question

My app is parsing a large http response, the http response is over 6 megabytes and is json, but not in a standard schema.

        final char[] buffer = new char[0x10000];
        StringBuilder out = new StringBuilder();
        Reader in = new InputStreamReader(is, "UTF-8");
        int read;
        System.gc();
        do
        {
            read = in.read(buffer, 0, buffer.length);
            if (read > 0)
            {
                out.append(buffer, 0, read);
            }
        } while (read >= 0);
        in.close();
        is.close();
        in = null;
        is = null;
        System.gc();
        return out.toString();

It doesn't matter if there is a bufferedreader from a file, or an inputstream, the StringBuilder simply cannot contain the entire object and it fails at out.append(buffer, 0, read); or it will fail at out.toString() as another copy may be made

IOUtils.copy from the apache library is doing the same things under the hood and it will also fail.

How can I read this large object in for further manipulation. Right now this method fails on Android 2.2 and 2.3 devices, and uses more memory than I want on newer devices.

Similar questions all have answers that involve appending to a stringbuilder, reading in lines, or have incomplete solutions that are only hints, and that doesn't work.

323go · Accepted Answer · 2013-08-28 14:41:42Z

4

You need to do one of two things:

Get multiple smaller JSON responses from the server and parse those. This might be preferable on a mobile device, as large chunks of data might not be transmitted reliably, which will cause the device to request the entire thing repeatedly.
Use a streaming JSON parser, such as Jackson, to process the data as it comes in.

answered Aug 28, 2013 at 14:41

323go

14.3k6 gold badges37 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

CQM Over a year ago

hm, regarding using Jackson, this would require me to rewrite my entire httpget method that is being used throughout the app. And secondly, does this mean for this one api call I will need an object with every single variable declared? It has nested objects too

CQM Over a year ago

well Jackson crashes on the same spot actually "java.lang.StringBuilder.toString(StringBuilder.java:663)" because it seems to use string builder internally too! hm

CQM Over a year ago

I went with GSON, thank you for pointing me in the right direction of streaming parsers

Collectives™ on Stack Overflow

android java not enough sequential memory for stringbuilder

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related