1

I want to ask, is it normal to handle OutOfMemoryError while doing batch-inserts?

I am using following code to batch-insert in mysql:

try
{
    Connection con = null;
    PreparedStatement ps = null;

    con = Manager.getInstance().getConnection();
    ps = con.prepareStatement("INSERT INTO" + 
    " movie_release_date_pushed_to_subscriber"
    + "(movie_id,cinema_id,msisdn,sent_timestamp)VALUES(?,?,?,?)");

    for (String msisdn : subscriberBatch)
    {
        try
        {
            ps.setInt(1, movieToBeReleased.getMovieId());
            ps.setInt(2, movieToBeReleased.getCinemaId());
            ps.setString(3, msisdn);
            ps.setTimestamp(4, new java.sql.Timestamp(new Date().getTime()));
            ps.addBatch();
        }
        catch (OutOfMemoryError oome)
        {
            ....
            ps.executeBatch();
        }
    }

    ps.executeBatch();
}
catch (Throwable e)
{
    ....
}
finally
{
    try
    {
        Manager.getInstance().close(ps);
        if (con != null)
        {
            con.close();
        }
    }
    catch (Throwable e)
    {
        ....
    }
}

NOTE: Any kind of advice/recommendation is most welcome,

3
  • Do you get a lot of OutOfMemoryErrors based on the size of the batch? Commented Feb 26, 2013 at 11:21
  • Not a lot, but I want to place the check just to be safe.(And I can't guess the appropriate batch-size, as I don't know on which machine will this app run) Commented Feb 26, 2013 at 11:22
  • 1
    I am afraid if there is no space in JVM then how can you do addition execute in same JVM Commented Feb 26, 2013 at 11:23

2 Answers 2

5

No its not normal. And your catch handler is totally ineffective. Catching an OOME does not miraculously solve its root cause - exhaustion of program memory. You get that error after the runtime has made a best effort at reclaiming memory, and failed. You should not be trying to execute code at that point, you may not even be able to log messages!

If you feel for whatever reason that your batch statement may cause an OOME, then you should either:

  • Break up the batch cycle into smaller 'buckets'
  • Make more memory available to the program
Sign up to request clarification or add additional context in comments.

3 Comments

Is there any way to know if JVM is near OOME, before it actually occurs?
@djaqeel - Sure. You can use a MemoryMXBean to receive notifications on your heap usage. Thats from within your program. You can also monitor the application externally, using a tool like JVisualVM (thats more for debugging than anything else).
If you could include MemoryMXBean into your answer, than it would be a more complete answer.
2

It makes no sense to try to execute the batch if you get an OutOfMemoryError. However, you can replace your INSERT query with an INSERT IGNORE INTO and in case of an OOME, ask the user to run the batch again after restarting JVM.

What INSERT IGNORE INTO will do is not run the insert if the primary key already exists in the table, so your batch will resume from where it crashed the app.

However, I will have to warn you that this is probably a very dirty way to circumvent this situation.

1 Comment

Keep in mind that it's a workaround. If you keep getting the OOME, then you might want to look into better memory management.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.