Android string concat giving outofmemoryerror

Question

I'm working on a project where I'm downloading packets of data over internet and parsing them in my application. Every packet has it's own structure and in the beginning I'm receiving only text data. But in some point I start to get binary packets with images and this kind of stuff and sometimes on some devices when the binary file is too big I'm getting this error. Actually it's never happend while I'm testing, but I'm getting some reports from users. Here is actually how I'm doing it :

reading response :

InputStream response = new BufferedInputStream(connection.getInputStream());

        int bytesRead = -1;
        byte[] buffer = new byte[30 * 1024];
        while ((bytesRead = response.read(buffer)) > 0 && stopThread) {
            byte[] buffer2 = new byte[bytesRead];
            System.arraycopy(buffer, 0, buffer2, 0, bytesRead);
            handleDataFromSync(buffer2);
        }

and parsing the data like this :

public void handleDataFromSync(byte[] buffer) {
    RPCPacket packet;
    String responseBody;

    while(!stopThread) return;
    try {
        responseBody = new String(buffer, "UTF-8");
        StringBuilder tmp = new StringBuilder(responseBody);
        totalBytesReceived += responseBody.length();

        if (tmpBuffer != null) {
            tmpBuffer = tmpBuffer.append(tmp);
            tmp = tmpBuffer;
        }
        int bufferLen = tmp.length();
        int lastLoc = 0;
        boolean gotPacket;
        boolean gotField;
        String thisPart = "";

        try {
            do {
                gotPacket = false;
                gotField = true;
                int needsSize = packetFieldSizes[tmpCurrentField - 1];
                if (tmpCurrentField == packetFieldSizes.length) {
                    needsSize = payloadSize;
                }
                if (needsSize > bufferLen - lastLoc) {
                    gotField = false;
                    String proba = tmp.substring(lastLoc);
                    tmpBuffer =  new StringBuilder(proba);
                    break;
                }
                thisPart = tmp.substring(lastLoc, lastLoc + needsSize);
                lastLoc += needsSize;
                if (gotField) {

                    switch (tmpCurrentField) {

                    case 1: {
                        long intVal = Long.parseLong(thisPart);
                        objectIdentificator = (int) intVal;
                        break;
                    }
                    case 2: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        pType = (short) intVal;
                        break;
                    }
                    case 3: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        operationType = (short) intVal;
                        break;
                    }
                    case 4: {
                        objectOId = thisPart;
                        break;
                    }
                    case 5: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        id = (int) intVal;
                        break;
                    }
                    case 6: {
                        long intVal = Long.parseLong(thisPart);
                        payloadSize = (int) intVal;
                        dataSize = (int) intVal;
                        break;
                    }
                    case 7: {
                        hashH = thisPart;
                        break;
                    }
                    case 8: {
                        long intVal = Long.parseLong(thisPart);
                        if (intVal == 0) {
                            isBad = true;
                            break;
                        }
                        dataType = (short) intVal;
                        break;
                    }
                    case 9: {
                        if (payloadSize != 0) {
                            byte[] tmpData = Base64.decode(thisPart);
                            first = tmpData;
                        }
                        break;
                    }
                    }

                    if (tmpCurrentField >= packetFieldSizes.length)
                        gotPacket = true;

                    if (gotPacket) {
                        Log.d("", "Gotpacket!");
                        packet = new RPCPacket(objectIdentificator,
                                RPCPacketType.getPacketTypeByValue(pType),
                                RPCOperationType.getByValue(operationType),
                                objectOId, id, dataSize, hashH,
                                RPCPacketDataType.getByValue(dataType),
                                first);
                        parseRPCPacket(packet);

                        myProgress++;
                        update();
                        Log.e("","myProgress : "+myProgress);
                        Log.e("","TOTAL PACKETS : "+RPCCommunicator.totalPackets);

                        // release temp fields
                        objectIdentificator = 0;
                        pType = 0;
                        operationType = 0;

                        objectOId = null;

                        id = 0;
                        dataSize = 0;

                        hashH = null;

                        dataType = 0;

                        first = null;

                        tmpCurrentField = 1;
                        payloadSize = 0;

                    } else {
                        tmpCurrentField++;
                    }
                }

                // you baad bad buffer
                assert (lastLoc <= bufferLen);

                if (isBad)
                    break;

            } while (true);

        } catch (IOException e) {
            e.printStackTrace();
            RPCCommunicator.writeLogs(e.toString(), "Synchronization" ,"handleDataFromSync");

        } finally {
            thisPart = null;
            tmp = null;
        }
    } catch (Exception e) {
        e.printStackTrace();
        RPCCommunicator.writeLogs(e.toString(), "Synchronization","handleDataFromSync");
    }

}

so the error is thrown in that line : tmpBuffer = tmpBuffer.concat(tmp); only when the image is too big and it's concatenating the string a few times to get the whole packet. I'm reading response in 30KB pieces, but I can receive images with 300, 400KB and etc.

So any kind of idea how can I get rid of this issue. I'm not really sure what can I use instead of this.

Thanks in advance!

Start by using a StringBuilder (or StringBuffer) instead of using Strings. Are you concat:ing the binary stuff into your tmp variable also? — Jens
– Jens, Commented Jan 31, 2012 at 8:34
no, I'm concatenating only strings, but I'm reading the response in byte array, which I'm using to create resposeBody — Android-Droid
– Android-Droid, Commented Jan 31, 2012 at 8:38
Well, if you give us a bit more info on the protocol in question (I assume it's pretty custom?) maybe we can see if there's a bit more efficient way to do it. — Jens
– Jens, Commented Jan 31, 2012 at 9:18
Ok, Ican show you how I'm parsing the packets. Just edited the question.. Actually I've just change it with StringBuilder...but can't try it because it never happened while I'm using the app. — Android-Droid
– Android-Droid, Commented Jan 31, 2012 at 10:00

Display name · Accepted Answer · 2012-01-31 08:41:59Z

3

Just like Jens said, just use a StringBuilder (http://developer.android.com/reference/java/lang/StringBuilder.html) or a StringBuffer (http://developer.android.com/reference/java/lang/StringBuffer.html) if you need synchronized calls from different threads.

In your current code every time you concat something to the String a new object is created, thus the memory problems can occur. When using a StringBuilder only one Object is used.

answered Jan 31, 2012 at 8:41

Display name

2,7453 gold badges33 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Jens · Accepted Answer · 2012-01-31 10:24:39Z

1

So, for a starter, the way you read your stream allocates too much memory, specifically this part:

byte[] buffer2 = new byte[bytesRead];
System.arraycopy(buffer, 0, buffer2, 0, bytesRead);
handleDataFromSync(buffer2);

Consider revising the API of handleDataFromSync(byte[] buffer) to handleDataFromSync(byte[] buffer, int start, int count) and do like this when you read you stream:

while ((bytesRead = response.read(buffer)) > 0 && stopThread) {
    handleDataFromSync(buffer, 0, bytesRead);
}

You can create a string from that using new String(buffer, offset, count, "UTF-8") in handleDataFromSync (just dumping bytes into a String in this way is not guaranteed to result in a properly decoded String if you including any character that will use more than one octet in UTF-8, such as ÅÄÖ or other junk).

The string handling in your parsing is a bit too vague to judge - have you measured / printed how much tmpBuffer grows when parsing for instance?

In your case I'd consider using an InputStreamReader, and trying to revise the parsing altogether - is there any reason you must read 30kb chunks?

(FYI, assert(..) is pretty much disabled in Android, check & throw an exception if you want to guard against stuff).

answered Jan 31, 2012 at 10:24

Jens

17.1k4 gold badges59 silver badges52 bronze badges

4 Comments

Android-Droid Over a year ago

I'm doing this System.arraycopy(buffer, 0, buffer2, 0, bytesRead); , because when I receive the last packet from the stream, it can be less than 30kb, and in that situation my buffer will be filled to it's size with some non characters...as I saw that before starting usin' this.

Android-Droid Over a year ago

and forget to mention, I'm reading the stream in 30kb chunks, because I've never know how much data I'll download. In some situations it can be like 5-10MB, in other it can be like 300MB too.

Jens Over a year ago

Well, you're not processing the received bytes - you're converting them into a String - and a string can be created from the "larger" 30 kb buffer if you supply an offset & count. The savings are not that great - less fragmentation and GC on the heap probably.

Android-Droid Over a year ago

I'll read your answer again carefully. Because actually in some cases when I'm downloading a big stream like 50MB it ca take line 2-3minutes to downloa it and to parse it..which I'm trying to avoid.

Collectives™ on Stack Overflow

Android string concat giving outofmemoryerror

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related