3

I built my first Java program which is built on top of the Interactive Brokers Java API. That may or may not be important. I just extended the main API classes with a couple new classes.

The program is making data queries to a remote server. When the server responds, I log the received data to a local MySQL data base. Once the program finishes logging the data, the program will make the next data request.

I am having a problem after leaving the program running for some time, after making a couple hundred server requests. I will see this error, then the program doesn't continue to execute:

java.lang.OutOfMemoryError: Java heap space

I did some research, and from what I read, I conclude that the program is creating many new variables, and not destroying old worthless ones. Since I am using Netbeans for development, I used the Netbeans profiler to inspect if this was the case. See the picture here:

enter image description here

After running the program for quite some time, more and more of the memory is used up by Byte. So it seems that my theory is still true.

I don't really know where to go from here. There is no reference to a class or specific variable, just a variable type. How can pinpoint where the problem is coming from?

UPDATE

I corrected a specific problem that was mentioned by BigMike in the comments. Previiously, I was creating many Statements in the JDBC MySQL Connector API, and I was calling .execute() to execute the statements, but I wasn't closing the statement with .close().

I made sure the add the statement.close() call after each execution, and the program runs much better now. By looking at the RAM usage for this program, it seems to solved the problem. I am also not seeing the Java heap space error anymore, which is nice.

Thanks!

Improve this question
6
  • 1
    you got 1 million byte [] (bytes arrays), maybe you're not releasing something. Commented Mar 14, 2013 at 15:35
  • 1
    IB apis seems to use sockets to send data, maybe you need to "release" resources after each operation. I don't know that api, thus this is mere speculation. Check also you're actually closing statements in the db logging code. Commented Mar 14, 2013 at 15:43
  • every line that is logged is logged with the same statement. I never "close" it, just re-define the statement variable with a new String... Commented Mar 14, 2013 at 15:45
  • so you're creating statement once, then using statement.execute() many times without using statement.close () ? Commented Mar 14, 2013 at 15:48
  • 1
    Add a close() ;) see blog.shinetech.com/2007/08/04/… for further informations. Commented Mar 14, 2013 at 15:52

2 Answers 2

Reset to default
2

It's very hard to say what might be wrong by simply that.

It might have to do with Streams that you are opening that aren't being closed when you no longer need them. Double check methods that allocate resources (reading from files, database, etc), especially if they read data into streams, and make sure you close those streams in a finally clause.

Apart from that, you can try and profile what methods are being called more often, etc, to try and narrow down the problem to a specific part of your code.

I found a site with a reasonable explanation of how Garbage Collection works, and what can cause OutOfMemoryErrors:

http://www.kdgregory.com/index.php?page=java.outOfMemory

If you read through that, there's a specific reference to high allocation of Object[] and byte[], that might point you in the right direction.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

thank you. I am still pretty new to Java. I have been doing a few try{}catch{} things to handle exceptions, but I have never put in a finally. I will look more into Streams also.
Updated with a link that you might find useful, a bit long, but worth it.
1

Generally speaking, this comes about for one of two reasons:

  1. There is a memory leak in the application, such that the application fails to release items for garbage collection, leading to the JVM running out of memory over time.

  2. The application attempted a one-off operation that would require more memory than is available, leading to the JVM running out of memory due to the operation.

Since your output seems to indicate that the bulk of the memory is consumed by literally a million plus small byte arrays, my guess is that #1 is probably the culprit; however, to verify this, restart your application and watch it's memory consumption over time. It will bounce up and down, but really you only need to watch the trend of consumption. If the consumption average continues to climb over time, you have a memory leak.

To solve this issue, you typically need the source code, and need to find the parts of the code where the troubling objects are being created, used, and then "stored" far beyond the last time that they will ever be used. The solution is to correct the code to no longer store them. HashMaps, Lists, and other Collections are often accomplices in memory leak problems.

If you lack the source code, you can attempt to measure the trend of the memory consumption, and schedule shutdowns and restarts of the application to effectively "reset the clock" such that you choose your downtime instead of watching the application choose it for you.

If it is a one-off operation (not likely considering your data) then you won't see an upward trend in memory consumption until the triggering event occurs. In such a case, with access to the source code, you should protect your application from processing data that grows very far outside of normal operating parameters. For example, reading a message from the network typically takes only a few KB, but in exceptional circumstances a client might transmit forever. In such a case, kill the message processing and throw the message away with an error if you exceed a maximum message size limit of 10 MB.

Without access to the source code in the latter scenario, the only hope is to identify the incoming upset, hunt down the source of the errant transmission, and attempt to manipulate it to prevent the overload of output.

The variations on how to approach these techniques are vast, but now you have a few ideas.

Improve this answer

3 Comments

I'd start by searching for the creation of new byte[]s in your code and finding what happens to them. Just store them in a Map? When do they get removed? Store them in a List? When do they get removed? Store them in a Class? When does it get removed from the Class, or when does the Class itself get removed? Eventually you'll find something that doesn't get removed, and that will be the culprit. Lots of tools out there to automate such a process, but you must understand the basic process to use them effectively.
Thanks Edwin, that's very helpful. Luckily it looks like the problem was coming from the code that I wrote, so it was easy to fix. It seemed to have fixed the byte problem, now it looks like the chars are growing faster than everything else... haha.
@jeffery_the_wind You're welcome. It's so nice that you found the initial problem. It's a pain, but you probably duplicated this kind of error a few times in the code. The good thing is that now you are aware of the issue, you've put a new skill in your pocket and will probably avoid the error in future projects. I love a good bug hunt, so bag a memory leak bug for me!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.