Still getting "OutOfMemoryError: Java heap space" exception even when increasing the JVM Heap Space to 8GB

Question

I have a snipper code below to upload a file to another storage. It works properly with files < 1GB of data. However, it exposes "OutOfMemoryError: Java heap space" exception with files size > 1GB (I verified with 1.2GB and 1.5GB)

After searching some posts, I increase the JVM Heap Space to 8GB. But, I still see the exception.

My machine is 16GB of RAM

The JVM version and Java Runtime Environment settings as below

Here is the snippet code:

protected boolean uploadFile(File file) throws BaseAFException {
    try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r");
         FileChannel channel = randomAccessFile.getChannel();
         FileLock lock = channel.lock(0, Long.MAX_VALUE, true)) {

        PayloadMessage payload = new PayloadMessage();


        try (ByteArrayOutputStream out = new ByteArrayOutputStream()) {
            int bufferSize = 1024;
            if (bufferSize > channel.size()) {
                bufferSize = (int) channel.size();
            }
            ByteBuffer buff = ByteBuffer.allocate(bufferSize);
            while (channel.read(buff) > 0) {
                out.write(buff.array(), 0, buff.position());
                buff.clear();
            }
            
            
            //the exception exposes before the debugger hits the line below         
            payload.setContent(out.toByteArray()); 
        }
        
        //Upload file will be handled here
        
        return true;
    } catch (OverlappingFileLockException e) {      
        return false;
    } catch (Exception e) {
        throw new Exception("Could not upload file " + file.getAbsolutePath(), e);
    }
}

The exception stack trace:

Throwable : java.lang.OutOfMemoryError: Java heap space java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
    at java.util.concurrent.FutureTask.report(Unknown Source)
    at java.util.concurrent.FutureTask.get(Unknown Source)
    at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.io.ByteArrayOutputStream.grow(Unknown Source)
    at java.io.ByteArrayOutputStream.ensureCapacity(Unknown Source)
    at java.io.ByteArrayOutputStream.write(Unknown Source)

I would greatly appreciate it if you kindly give me some advice!

In your code, it looks like you are trying to read the file into the byte array and store each byte in the memory before sending the file. For such big files, this is a bad technique because of the error you are currently getting. You should use streaming to send chunks of bytes to the destination without storing the whole file in the memory. Otherwise, you will not be able to send the file of size larger than the size of your free RAM. Remember, that the size of the file does not always imply that in a Java byte array it will be stored as the same size in RAM. — Markiian Benovskyi
– Markiian Benovskyi, Commented Aug 25, 2021 at 10:21
You have multiple copies of file in memory simultaneously. I suggest you to change the way you upload file. — talex
– talex, Commented Aug 25, 2021 at 10:57

Holger · Accepted Answer · 2021-08-25 14:35:48Z

Note that using a RandomAccessFile just to get a FileChannel is obsolete since Java 7.

You copy the file’s content into a ByteArrayOutputStream. Once you finished this operation, the ByteArrayOutputStream will have an array at least as large as the copied data, but since the final size isn’t known when the internal capacity needs to be increased, it may be up to two times the needed size.

Then, you call toByteArray() on it, whose documentation says “Creates a newly allocated byte array.”.

So after returning from toByteArray(), you are using heap memory between two and three times the actual file size.

The ByteArrayOutputStream holding the entire file but potentially having up to twice the necessary capacity
The array returned by toByteArray() matching the file size

So when reading a file of 1.5GB this way, you may easily occupy up to 4.5GB only for these byte arrays. Well, it’s not the exact truth, as array sizes are limited to roughly 2GB, so the array size will not exceed this, but it’s still 2GB for the internal array and 1.5GB for the new array.

Note that even while copying into the ByteArrayOutputStream, the allocated memory can be much higher. In the worst case, the stream may already hold an array of almost 1.5GB and have to increase it, so it will allocate a ~2GB array, copy the data, and only afterwards it may drop the old array. So it will already occupy 3.5GB temporarily when growing the array.

You should recheck the APIs you’re using, whether they truly require you to pass a byte array holding the entire file in heap memory, instead of e.g. a channel to copy from or a ByteBuffer which could be a memory mapped file instead of encapsulating an array.

If that’s not possible, eliminate the redundant copies at least.

You may omit the ByteArrayOutputStream, just use a ByteBuffer of the file’s size, read the file into it, and use its byte array.

protected boolean uploadFile(File file) throws BaseAFException {
    PayloadMessage payload = new PayloadMessage();

    try(FileChannel channel = FileChannel.open(file.toPath(), StandardOpenOption.READ);
        FileLock lock = channel.lock(0, Long.MAX_VALUE, true)) {

        long size = channel.size();
        if(size > Integer.MAX_VALUE) {
            throw new Exception("can't keep " + size + " bytes in an array");
        }

        ByteBuffer buff = ByteBuffer.allocate((int)size);
        do channel.read(buff); while(buff.hasRemaining());

        payload.setContent(buff.array());
    }
    catch( … ) …

    // handle upload HERE when resources have been closed
    // to have at least one advantage of reading the entire file into memory
}

Or just use built-in features

PayloadMessage payload = new PayloadMessage();
payload.setContent(Files.readAllBytes(file.toPath()));

Though this may not use locking while reading.

Collectives™ on Stack Overflow

Still getting "OutOfMemoryError: Java heap space" exception even when increasing the JVM Heap Space to 8GB

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related