Java collection and memory optimization

Question

I wrote a custom index to a custom table which uses 500MB of heap for 500k strings. Only 10% of the strings are unique; the rest are repeats. Every string is of length 4.

How i can optimize my code? Should I use another collection? I tried to implement a custom string pool to save memory:

public class StringPool {

    private static WeakHashMap<String, String> map = new WeakHashMap<>();

    public static String getString(String str) { 
        if (map.containsKey(str)) {
            return map.get(str);
        } else {
            map.put(str, str);
            return map.get(str);
        }
    }
}

private void buildIndex() {
        if (monitorModel.getMessageIndex() == null) {
            // the index, every columns create an index
            ArrayList<HashMap<String, TreeSet<Integer>>> messageIndex = new ArrayList<>(filterableColumn.length);
            for (int i = filterableColumn.length; i >= 0; i--) {
                // key -> string,   value -> treeset, the row wich contains the key
                HashMap<String, TreeSet<Integer>> hash = new HashMap<>();
                messageIndex.add(hash);
            }
            // create index for every column
            for (int i = monitorModel.getParser().getMyMessages().getMessages().size() - 1; i >= 0; --i) {
                TreeSet<Integer> tempList;

                for (int j = 0; j < filterableColumn.length; j++) {
                    String value  = StringPool.getString(getValueAt(i, j).toString());
                    if (!messageIndex.get(j).containsKey(value)) {
                        tempList = new TreeSet<>();
                        messageIndex.get(j).put(value, tempList);
                    } else {
                        tempList = messageIndex.get(j).get(value);
                    }

                    tempList.add(i);
                }
            }
            monitorModel.setMessageIndex(messageIndex);
        }
    }

500,000 4 character strings is only a few dozen megabytes of memory with no caching at all. Think you're looking in the wrong place. — Affe
– Affe, Commented Aug 15, 2012 at 17:30
I agree with Affe, that should not exceed a couple of MB, even assuming 50 Bytes per 4 letter String (which is pessimistic) would only get you to 25MB. — G. Bach
– G. Bach, Commented Aug 15, 2012 at 17:38
ArrayList<HashMap<String, TreeSet<Integer>>> -- Wow, that a structure! :) You impose huge overhead using such data structure. It very well can be a reason of the high memory consumption, not Strings themselves. I have written a blog post some time ago about Java Collection overhead: plumbr.eu/blog/fat-collections — Nikem
– Nikem, Commented Aug 16, 2012 at 7:33
Thanks, this kind of answer i search, i will read your blog. — flatronka
– flatronka, Commented Aug 16, 2012 at 10:46

Dev · Accepted Answer · 2012-08-15 17:21:13Z

5

No need to come up with a custom pool. Just use String.intern().

answered Aug 15, 2012 at 17:21

Dev

12.3k3 gold badges45 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

flatronka Over a year ago

Thanks your answer, i try this but it's didn't work, reduce the heap only -2MB.

Dev Over a year ago

This would save you the trouble of creating your own string pool. I do believe @AaronD answer is more on point though, a lot of your memory usage may be coming from the number of nested data structures being instantiated.

AaronD · Accepted Answer · 2012-10-26 17:27:31Z

You might want to examine your memory heap in a profiler. My guess is that the memory consumption isn't primarily in the String storage, but in the many TreeSet<Integer> instances. If so, you could optimize considerably by using primitive arrays (int[], short[], or byte[], depending on the actual size of the integer values you're storing). Or you could look into a primitive collection type, such as those provided by FastUtil or Trove.

If you do find that the String storage is problematic, I'll assume that you want to scale your application beyond 500k Strings, or that especially tight memory constraints require you to deduplicate even short Strings.

As Dev said, String.intern() will deduplicate Strings for you. One caveat, however - in the Oracle and OpenJDK virtual machines, String.intern() will store those Strings in the VM permanent-generation, such that they will not be garbage-collected in the future. That's appropriate (and helpful) if:

The Strings you're storing do not change throughout the life of the VM (e.g., if you read in a static list at startup and use it throughout the life of your application).
The Strings you need to store fit comfortably in the VM permanent generation (with adequate room for classloading and other consumers of PermGen). Update: see below.

If either of those conditions is false, you are probably correct to build a custom pool. But my recommendation is that you consider a simple HashMap in place of the WeakHashMap you're currently using. You probably don't want these values to be garbage-collected while they're in your cache, and WeakHashMap adds another level of indirection (and the associated object pointers), increasing memory consumption further.

Update: I'm told that JDK 7 stores interned Strings (String.intern()) in the main heap, not in perm-gen, as earlier JDKs did. That makes String.intern() less risky if you're using JDK 7.

Thanks your answer Aaron, i try to work with primitive collection, i try String.intern(), but my heap reduces more memory.

Collectives™ on Stack Overflow

Java collection and memory optimization

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related