Java - byte buffer Out Of Memory

Question

Sort of new to Java but I've been able to learn quite a bit quite quickly. There are still many methods that elude me though. I'm writing a Java program that is to run through a bunch of files and test them thoroughly to see if they are a valid level pack file. Thanks to Matt Olenik and his GobFile class (which was only meant to extract files) I was able to figure out a good strategy to get to the important parts of the level pack and their individual map files to determine various details quickly. However, testing it on 37 files (5 of which aren't level packs), it crashes after testing 35 files. The 35th file is a valid level pack. The error it gives is:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

     Iterator iter = itemList.keySet().iterator();
     numJKLs = 0;
     while (iter.hasNext()) {
        String nextFile = (String) iter.next();
        if (nextFile.startsWith("jkl\\")) {
           System.out.println(((ItemInfo) itemList.get(nextFile)).length);
-->        byte[] buffer = new byte[((ItemInfo) itemList.get(nextFile)).length];
           gobRaf.seek(((ItemInfo) itemList.get(nextFile)).offset);
           gobRaf.read(buffer, 0,
                 ((ItemInfo) itemList.get(nextFile)).length);
           String[] content = new String(buffer).toLowerCase().split(
                 "\n");
           levels.add(numJKLs, new JKLFile(theFile, nextFile, content));
           gametype = levels.get(numJKLs).getFileType();
           progress.setProgText(nextFile);
           buffer = null;
           content = null;
           numJKLs++;
        }
     }

Arrow shows where the error is marked. The ´JKLFile´ class reads the content array for these important parts, but should (in theory) dispose of it when done. Like here, I set content = null; in JKLFile just to make sure it is gone. If you are wondering how big the map file it stopped on is, well, it managed to pass a 17 Mb map file, but this one was only 5 Mb.

As you can see these JKLFile objects are kept in this object (GobFile.java) for easy access later on, and the GobFile objects are kept in another class for later access, until I change directory (not implemented yet). But these objects shouldn't be too packed with stuff. Just a list of file names and various details.

Any suggestions on how I can find out where the memory is going or what objects are using the most resources? It would be nice to keep the details in memory rather than having to load the file again (up to 2 seconds) when I click on them from a list.

/Edward

You're adding all your files to levels? How big do you suppose that gets???? — Hot Licks
– Hot Licks, Commented Mar 14, 2014 at 17:02
@HotLicks I'm not adding the whole thing to levels. True, I'm sending the contents, but I was hoping that it would not get saved. Just process the key details and save 2 ints and 2 strings (level name and file name). — Edward
– Edward, Commented Mar 14, 2014 at 19:21
It looks to me like you're adding the contents of each file to levels -- levels.add(numJKLs, new JKLFile(theFile, nextFile, content));. So you're effectively reading all the files into storage at once (only the way you read them they take up probably 5x as much space as the original files). — Hot Licks
– Hot Licks, Commented Mar 14, 2014 at 19:25
@HotLicks Well, the details I'm after from a particular file in the whole level pack are in the middle of the file with an unknown number of declarations before it and an unknown number of stuff after it. I tried with .readLine() but it took quite a while to run through some files. This has been the fastest strategy yet. — Edward
– Edward, Commented Mar 14, 2014 at 19:43
Then you must deal with the consequences of filling up the heap with all that data. You have every line of every file in your levels object, apparently. — Hot Licks
– Hot Licks, Commented Mar 14, 2014 at 20:46

indivisible · Accepted Answer · 2014-03-14 17:11:56Z

1

Right now you are creating a new buffer for every loop. Depending on your garbage collection some of these may not get recycled/freed from memory and you are then running out. You can call the garbage collector using System.gc() but there is no guarantee that it actually runs at any given time.

You could increase the memory allocated to the JVM on the command line when calling your program with the -Xmx and -Xms arguments. Also if you are not on 32bit windows ensure you are using a 64bit JVM as the 32bit has memory limitations that the former does not. Run java -version from the command line to see the current system JVM.

You can move the declaration for your byte[] buffer outside your loop (above your while) so that it gets cleared and reused on each iteration.

Iterator iter = itemList.keySet().iterator();
numJKLs = 0;
byte[] buffer;
while (iter.hasNext()) {
    ...
    buffer = new byte[] ...
    ...
}

How big are the files you are handling? Do you really need to add every one of them to your levels variable? Could you make do with some file info or stats instead? How much RAM has your JVM currently got allocated?

To look at the current memory usage at any given time you will need to debug your program as it runs. Look up how to do that for your IDE of choice and you will be able to follow any Object's current state through the program's lifespan.

edited Mar 14, 2014 at 17:11

answered Mar 14, 2014 at 16:56

indivisible

5,0304 gold badges33 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Hot Licks Over a year ago

Running System.gc() will do nothing to prevent the error. Increasing heap size simply hides the fact that he's being exceedingly wasteful of storage by reading in all the files at once.

Edward Over a year ago

Well, I'm not actually adding the whole content to my levels variable. True, I am trying to send the content to the class, but didn't think it would be saved. Just process the key details and save 2 ints and 2 strings (level name and file name). However, your answer did help me to reorganize my variables. Moving the byte[] buffer as well as the String[] content and some other variables outside the loops made the program complete all 37 files. Must try and remember this. Thanks.

Collectives™ on Stack Overflow

Java - byte buffer Out Of Memory

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related