Byte[] and java.lang.OutOfMemoryError reading file by bits

Question

I am trying to write a reader which reads files by bits but I have a problem with large files. I tried to read file with 100 mb and it took over 3 minutes but it worked.

However, then I tried file with 500 mb but it didn't even started. Because of this line:

byte[] fileBits = new byte[len];

Now I am searching for sulutions and can't find any. Maybe someone had solved it and could share some code, tips or idea.

if (file.length() > Integer.MAX_VALUE) {
    throw new IllegalArgumentException("File is too large: " + file.length());
}

int len = (int) file.length();
FileInputStream inputStream = new FileInputStream(file);

try {
    byte[] fileBits = new byte[len];
    for (int pos = 0; pos < len;) {
        int n = inputStream.read(fileBits, pos, len - pos);
        if (n < 0) {
            throw new EOFException();
        }
        pos += n;
    }

inputStream.read(fileBits, 0, inputStream.available());
inputStream.close();

Do you really need the entire file at once? What prevents you from streaming it? — josh.trow
– josh.trow, Commented Feb 29, 2012 at 19:02
Did you try searching for OutOfMemoryError? See java -mx — Miserable Variable
– Miserable Variable, Commented Feb 29, 2012 at 19:04
Each index you create is 4 bytes. 500 million of them will try to allocate 2G of space. You can't solve it this way without more memory or as josh.trow suggested buffer/streaming — John Vint
– John Vint, Commented Feb 29, 2012 at 19:05
The solution that worked for us was to use a machine with 96GB RAM (with appropriate -XMx option) instead of 24GB when our app ran out of memory.. :D.. seriously, though answer what Josh said.. — Kashyap
– Kashyap, Commented Feb 29, 2012 at 19:06
@JohnVint how would creating a byte array use more than one byte per index? — mgibsonbr
– mgibsonbr, Commented Feb 29, 2012 at 19:12

Peter Lawrey · Accepted Answer · 2012-02-29 19:13:59Z

7

I suggest you try memory mapping.

FileChannel fc = new FileInputStream(file).getChannel();
MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_ONLY, 0, (int) fc.size());

This will make the whole file available almost immediately (about 10 ms) and uses next to no heap. BTW The file has to be less than 2 GB.

answered Feb 29, 2012 at 19:13

Peter Lawrey

535k83 gold badges770 silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

mgibsonbr · Accepted Answer · 2012-02-29 19:10:55Z

2

If you really need to load the entire file to memory at once, I can only suggest increasing the memory available to Java. Try invoking your program using the Xmx (max heap size) or Xms (initial heap size) arguments (use the latter if you know beforehand how much memory you'll need, otherwise the former might be best).

java -Xms512m -Xmx1g BigApp

As an alternative, you can use NIO's Memory-mapped files.

answered Feb 29, 2012 at 19:10

mgibsonbr

22.1k7 gold badges73 silver badges115 bronze badges

Comments

Tiago Pasqualini · Accepted Answer · 2012-02-29 19:10:49Z

1

You shouldn't open the whole file into memory. You need to create a byte array buffer with a fixed size, then you open your file from chunks of the size you defined.

answered Feb 29, 2012 at 19:10

Tiago Pasqualini

8215 silver badges12 bronze badges

Comments

aioobe · Accepted Answer · 2012-02-29 19:11:50Z

1

I suggest you use a RandomAccessFile.

answered Feb 29, 2012 at 19:11

aioobe

423k115 gold badges831 silver badges844 bronze badges

Collectives™ on Stack Overflow

Byte[] and java.lang.OutOfMemoryError reading file by bits

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related