delete memory allocated to a byte array

Question

I am receiving records in while loop at a socket server. where each records has a msg type followed by message length and actual message content.

The problem is that since I get around a million records and each record has a record size of 277 bytes. So, after around 40000 records, I get OutOfMemory error. The code flow looks something like this:

while (true) {              
            msgType = dIn.readByte();

            int msgIntType = msgType & 0xff;

                  // get message length

                  int msgIntLen = dIn.readInt();
                  if (msgIntLen != 0) {

     msgContent = new byte[msgIntLen];
                   switch(msgIntType) {
            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent);
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                }               
                break;

}

I resolved the OutOfMemory issue by explicitly calling System.gc() after sending ACK after every 2000 records processed and now it works perfectly fine and able to handle 1 million records without any error in less than 10 minutes. The modified code for case statement for calling System.gc() looks like:

            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent);
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                             System.gc();
                }               
                break;

But I read here on some other posts that calling System.gc() is not a good design approach ? Is it so ? If yes, could you guys suggest me some other way to get rid of this OutOfMemory error ?

Thanks in advance -JJ

EDIT: logic for processData():

public void processData(byte[] msgContent) throws Exception {

    InputStreamReader inp = new InputStreamReader(

            new ByteArrayInputStream(msgContent));

    BufferedReader br = null;
    try {

        br = new BufferedReader(inp);
                             String line;
        while ((line = br.readLine()) != null) {

                             process each line
                             .
                             }
                  } catch (Exception e) {
        logger.error("exception in " + Utils.getExecutingMethodName(e) 
                + " :" + e.getMessage());
    } finally {
        try {
            if (br != null)
                br.close();
        } catch (IOException e) {
            logger.error("Error: " + e);
        }
    }
}

What does processData() do? Is it possible it keeps a reference to the byte array somehow? — matt b
– matt b, Commented Apr 27, 2011 at 16:42
I suggest you take a heap dump of the out of memory error and analyse it. It should show wheer your memory is being held. It might point to a solution. — Peter Lawrey
– Peter Lawrey, Commented Apr 27, 2011 at 16:48
@Peter - could you tell me how to do this ? This might be really helpful. And we take this dump after program has failed ? ( I guess :) — user656189
– user656189, Commented Apr 27, 2011 at 16:56
Add the option -XX:+HeapDumpOnOutOfMemoryError and load the dump produced in a tool like YourKit. I think VisualVM can load it too. — Peter Lawrey
– Peter Lawrey, Commented Apr 27, 2011 at 16:58
Nothing to do with the problem, but there is no need for the finally block there (or even the close as there is no actual resource involved. (I think it was possibly a mistake to put close on InputStream and Reader.) — Tom Hawtin - tackline
– Tom Hawtin - tackline, Commented Apr 27, 2011 at 21:45

Tom Hawtin - tackline · Accepted Answer · 2011-04-27 16:41:39Z

2

Are you failing to close some resource and relying upon the finaliser thread to pick them up? Or have you just added a finaliser (probably unnecessary) that is preventing a significant amount of memory to be freed promptly.

answered Apr 27, 2011 at 16:41

Tom Hawtin - tackline

148k30 gold badges222 silver badges314 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

matt b Over a year ago

for example in the update that shows what processData() does, you should really close the InputStreamReader

Jay · Accepted Answer · 2011-04-27 16:44:12Z

2

If that is literally the only change you made, it is difficult to see how that would fix the problem. Whenever the Java Virtual Machine is running low on memory, it runs the garbage collector automatically before throwing an out of memory exception. There should be no need to do this yourself nor any value in doing so.

The only real solution to the problem you describe is to make sure that you clear out any references to objects that are no longer needed. Like if you say:

byte[] ba=new byte[bignumber];
process(ba);

and then you go on and do other things, ba is still sitting there hogging up memory. You want to either exit the function where it was defined, or set ba=null to lose the reference. Then the gc can recycle the memory.

answered Apr 27, 2011 at 16:44

Jay

27.6k12 gold badges68 silver badges116 bronze badges

2 Comments

user656189 Over a year ago

I tried ba = bull before break in my case statement and it did not help. System.gc() solved the problem for me.

Jay Over a year ago

Are you absolutely certain that you literally just added that one line and then the program worked? Because according to the documentation, that's impossible. Frankly this sounds like saying that you typed x=2+2 and x ended up being 5. I guess it's not literally impossible, there could be a bug in Java or some subtlety in the specs that I'm missing. But I've seen emphasized over and over JVM descriptions that the gc runs automatically when memory is low, and calling gc() is NEVER necessary. I would take your program as it is now, delete that gc call, CHANGE NOTHING ELSE, and see if it fails.

Java Drinker · Accepted Answer · 2011-04-27 17:28:09Z

0

What is the the type of your dIN variable? Maybe I am misunderstanding, but do you really need to read your input into a byte array, then consider the byte array as a stream, and then read the stream line by line? If you already know the structure of your content, why create all the intermediate steps. You could just as well process(dIn) somehow.

Also, just to confirm, is this running in a multi-threaded environment??

answered Apr 27, 2011 at 17:28

Java Drinker

3,1571 gold badge23 silver badges19 bronze badges

1 Comment

user656189 Over a year ago

right onw it is not running in multi-threaded env. I am reading my input in byte array as I recevied msg as a byte array where first byte is message type following with 4 bytes of msg length and followed by msg content bytes. I read msg type and msg length and then pass byte array content which is an ascii string. Now in process(), i rcv this byte array and do steps wrttn above in code for this func and add it to a stringbuilder obj. When these records(i.e. multiple process() calls)reach a buffer size we want, we then use the stringbuilder output to dump this buffer to a file.

Liv · Accepted Answer · 2011-04-28 09:44:53Z

0

Is there a maximum size to the data you receive (or can you enforce one)? In which case you can declare your byte array outisde the while and reuse it at each iteration without allocating more memory:

...
private static final int BUFFER_SIZE = 102400; //start with a buffer big enough to lower the chances of resizing it -- e.g. 100K
...
msgContent = new byte[BUFFER_SIZE];
while (true) {              
            msgType = dIn.readByte();

            int msgIntType = msgType & 0xff;

                  // get message length

                  int msgIntLen = dIn.readInt();
                  if (msgIntLen != 0) {
                   if( msgIntLen > msgContent.length ) //only resize when needed otherwise reuse
                     msgContent = new byte[msgIntLen];

                   switch(msgIntType) {
            case 4:
            //case MSG_DATA:
                // MSG_DATA
                recordCount++;
                processData(msgContent, msgIntLen); //need to change your function to also pass in the size of the message read!
                if (recordCount == 2000) {
                sendACK(dOut, msgIntType);
                logger.info("sent ACK for MSG_DATA");
                recordCount = 0;
                }               
                break;

}

edited Apr 28, 2011 at 9:44

answered Apr 27, 2011 at 16:46

Liv

6,1441 gold badge24 silver badges31 bronze badges

2 Comments

user656189 Over a year ago

I tried this solution also but we are running this JVM for different sizes of data stream where we don't know buffersize before hand.

Liv Over a year ago

you could adapt it to extend the buffer as needed so your buffer will extend only when the buffer size needs extending and otherwise just reuse the already allocated buffer. I'm about to edit my answer to show you how.

Collectives™ on Stack Overflow

delete memory allocated to a byte array

4 Answers 4

1 Comment

2 Comments

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related