While writing huge data to a file using java, how can one avoid java heap memory out of space error from occuring?..This is causing some failures for my program..
Regards,
magggi
-
3Show your code we may be able to help you to correct it.Colin Hebert– Colin Hebert2010-09-09 18:53:01 +00:00Commented Sep 9, 2010 at 18:53
-
I basically append all my content to a string buffer and then finally write & flush. This when data content is huge results in heap memory out of space errorMagggi– Magggi2010-09-09 18:54:44 +00:00Commented Sep 9, 2010 at 18:54
-
1Can you write to your file stream as you are generating the content? In other words, instead of appending to a string can you just write to your output stream? This is the typical solution -- you don't want to use a whole lot of memory unless you absolutely need to.dlongley– dlongley2010-09-09 19:00:06 +00:00Commented Sep 9, 2010 at 19:00
-
If you absolutely must use StringBuffer for some reason, you can write each list's formatted data to it, flush its contents to your OutputStream, and then clear the StringBuffer for the next iteration. This should work unless one list's formatting depends on the next list.dlongley– dlongley2010-09-09 19:12:13 +00:00Commented Sep 9, 2010 at 19:12
Add a comment
|
5 Answers
Your problem is that you're using a single StringBuffer to handle all your content. You should try to flush your data more often. You will gain performance and memory.
Without the original code I can't help you more, but try to flush your buffer from time to time.
Not advised in your case
If you still want to use your StringBuffer and flush it only once, then you'll have to increase your heap space with the -Xmx***M option on the java command.
Resources :
- oracle.com - basic IO
- oracle.com - Java command line tool (see -Xmx part for heap)
On the same topic :
Comments
From your comments it sounds like you're iterating over some objects that fill up a StringBuffer and, once you're done iterating, you flush the entire StringBuffer to a file. Instead of doing this, you can either write directly to an output stream on each iteration (recommended) or, on each iteration, write to your StringBuffer, flush it to the output stream, and then clear the StringBuffer for the next iteration (not preferred). This should lower your memory usage.
I don't recommend increasing the heap size -- it doesn't sound like the right solution to this problem.
Comments
Use the -Xmx parameter when launching the java program to set the maximal heap size:
java -Xmx256M com.foo.MyClass
2 Comments
Colin Hebert
If the file is that huge, allocating more memory isn't the best way to deal with it.
gawi
Note that the initial statement was rather vague. I tend to give short and simple answers when the context is not well defined. It was not even mentioned that he could modify the program. I've observed that StackOverflowers will always prefer the "cleanest" solution (disregarding the complexity) over simple, straight-forward, but non-scalable/slow/occasionally-incorrect solution. When the context of the question is not well-defined, maybe I should always give 2 answers: the hack and the good-practice.
Did you try passing the memory parameters to the applicaiton when starting it?
For example the following sets the maximum memory to 1 GB
java -Xms512m -Xmx1024m myClass
Comments
A couple of options
Increase Java Heap Size using the -Xmx JVM parameter. After -Xmx, write the number of megabytes your program needs to successfully run.
Use Streaming. Compute the next N bytes in the file and then write them using your FileOutputStream, throwing them away before computing the next N bytes. More code would be necessary to know if and how this would work for your program.
Edit: From what you are saying, it sounds like you could wrap the FileOutputStream in a BufferedOutputStream, drop the StringBuffer entirely and just write directly to the BufferedOutputStream.
