Memory usage in loading a 226MB text file

Question

I have to read a text file of 226mb made like this:

the first number is a index, the second a value. I want to load a vector of short reading this file (8349328 positions), so I wrote this code:

    Short[] docsofword = new Short[8349328];

        br2 = new BufferedReader(new FileReader("TermOccurrenceinCollection.txt"));             
        ss = br2.readLine();
        while(ss!=null)
        {
            docsofword[Integer.valueOf(ss.split("\\s+")[0])] = Short.valueOf(ss.split("\\s+")[1]);  //[indexTerm] - numOccInCollection
            ss = br2.readLine();
        }
    br2.close();

It turns out that the entire load takes an incredible amount of memory of 4.2GB. Really i don't understand why, i expected a 15MB vector. Thanks for any answer.

How are you determining the memory usage? Bear in mind that Short is not the same as short; there is an extra reference (pointer) for every element. — Oliver Charlesworth
– Oliver Charlesworth, Commented Apr 7, 2017 at 16:57

Durandal · Accepted Answer · 2017-04-07 18:00:39Z

3

There are multiple effects at work here.

First, you declared your array as type Short[] insted of short[]. The former is a reference type, meaning each value is wrapped into an instance of Short, consuming the overhead of a full blown object (most likely 16 bytes instead of two). This also inflates each array slot from two bytes to the reference size (generally 4 or 8 bytes, depending on heap size and 32/64 bit VM). The minimum size you can expect for the fully populated array is thus approximately: 8349328 x 20 = 160MB.

Your reading code is happily producing tons of garbage objects - you are using again a wrapper type (Integer) to address the array where a simple int would do. Thats at least 16 bytes of garbage where it would be zero with int. String.split is another culprit, you force the compilation of two regular expressions per line, plus create two strings. Thats numerous short lived objects that become garbage for each line. All of that could be avoided with a few more lines of code.

So you have a relatively memory hungry array, and lots of garbage. The garbage memory can be cleaned up, but the JVM decides when. The decision is based on available maximum heap memory and garbage collector parameters. If you supplied no arguments for either, the JVM will happily fill your machines memory before it attempts to reclaim garbage.

TLDR: Inefficient reading code paired with no JVM parameters.

answered Apr 7, 2017 at 18:00

Durandal

20.2k4 gold badges39 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Oliver Charlesworth Over a year ago

Great answer, though I question "JVM happily fills your machine's memory" - there is always a default Xmx value.

Descanso7 Over a year ago

Thanks @Durandal, you never stop learning, thanks to you i understood my errors. I found the Short-short error after posting this question, but I was completely unaware of how much garbage produces String.split, thanks.

Jitendra Kumar. Balla · Accepted Answer · 2017-04-09 04:22:43Z

0

If file is generated by you, use objectOutputStream, It very easy way to read the file.

As @Durandal, change the code accordingly. I am giving sample code below.

short[] docsofword = new short[8349328];

    br2 = new BufferedReader(new FileReader("TermOccurrenceinCollection.txt"));             
    ss = br2.readLine();
    int strIndex, index;
    while(ss!=null)
    {
       strIndex = ss.indexOf( ' ' );
       index = Integer.parseInt(ss.subStr(0, strIndex));
       docsofword[index] = Short.parseShort(ss.subStr(strIndex+1));
       ss = br2.readLine();
    }
br2.close();

Even you can optimise further. Instead of indexOf() we can write our own method, when char is matching to space, parse string as integer. After that we will get indexOf Space and index for get remain string.

edited Apr 9, 2017 at 4:22

answered Apr 7, 2017 at 18:36

Jitendra Kumar. Balla

1,2131 gold badge10 silver badges15 bronze badges

3 Comments

Jitendra Kumar. Balla Over a year ago

@user3329403 Check and let me know, so that I'll give more suggestions/code

Descanso7 Over a year ago

Thanks @Jitendra Kumar. Balla, i understood my errors. I found the "Short-short" error just after posting the question, but I was unaware of how much garbage produces String.split. Your code has some indexing error (substring last index is exclusive, so it doesn't need -1), but it works well , thanks you

Jitendra Kumar. Balla Over a year ago

@user3329403 Thanks for reply. Can you accept solution once its resolved.

Collectives™ on Stack Overflow

Memory usage in loading a 226MB text file

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related