Out of memory error while read a file to byte array using apache io utils

Question

I have created a class to generate checksum for the files using apache file utils and bouncy castle to generate checksum. I have read the given file using apache's - IOUtils and I have used bouncy castle's - bcprov to generate checksum.

The code which I used as below :

byte[] digest = null;
String result = "";

SHA1.Digest sha1 = new SHA1.Digest();
sha1.update(IOUtils.toByteArray(new FileInputStream(file)));
digest = sha1.digest();

for (int i = 0; i < digest.length; i++)
{
     result += Integer.toString((digest[i] & 0xff) + 0x100, 16).substring(1);
}

When I try to read a large size of file (i.e about 80MB file) I get out of memory space error as shown below

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteAr rayOutputStream.java:322)
        at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:463)
        at com.test.hashgenerator.HashGenerator.getSHA1CheckSum(HashGe nerator.java:230)
        at com.test.hashgenerator.HashGenerator.main(HashGenerator.jav a:319)

Is I need to enhance my code? Any help would be appreciated.

how much heap space are you giving your JVM? regardless, 80 meg is a huge amount of memory to try to allocate as a byte array. i suggest you push it through the digest in smaller chunks — slipperyseal
– slipperyseal, Commented Dec 2, 2015 at 6:12
Read and update in chunks of small buffers and increase Xmx size — Ravindra babu
– Ravindra babu, Commented Dec 2, 2015 at 9:08

Community · Accepted Answer · 2017-05-23 12:22:50Z

1

Your code crash exactly because of IOUtils.toByteArray(new FileInputStream(file)), SHA doesnt matter.

For more informations, see this for example: Most Robust way of reading a file or stream using Java (To prevent DoS attacks)

This code works fine on a 200 MB file, thanks to @SlipperySeal and this: Java: How to create SHA-1 for a file?

public byte[] createSha1(File file) throws Exception  {
MessageDigest digest = MessageDigest.getInstance("SHA-1");
InputStream fis = new FileInputStream(file);
int n = 0;
byte[] buffer = new byte[8192];
while (n != -1) {
    n = fis.read(buffer);
    if (n > 0) {
        digest.update(buffer, 0, n);
    }
}
return digest.digest();
}

edited May 23, 2017 at 12:22

CommunityBot

11 silver badge

answered Dec 2, 2015 at 11:46

guillaume girod-vitouchkina

3,0891 gold badge14 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Out of memory error while read a file to byte array using apache io utils

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related