We are trying to compare 2 CSV files which are of 3 gb size. We are receiving java heap memory error while trying to read these files with CSV map reader.can anybody please help me to solve this issue.
-
2Please provide your codeDaniel– Daniel2016-09-03 13:47:53 +00:00Commented Sep 3, 2016 at 13:47
-
If you are reading the entire contents of the CSV file into memory at once then you may encounter errors if you do not have enough memory to store the contents. Have you tried reading writing your program to read each file a line at a time and compare them? or write it to store 1 line from a file and search the entire other file for that content?D3181– D31812016-09-03 13:53:22 +00:00Commented Sep 3, 2016 at 13:53
Add a comment
|
1 Answer
One option is to expand the memory for the heap. The initial heap size is 128 megs by default. Java has a flag you can pass it to increase this value.
You can initialize the heap size with -Xms[size] or you can give the JVM a maximum heap size with -Xmx[size].
In your case I would use -Xms to give the program more memory. For instance if you want the heap size to start out at 512 MB then when running the program you would run:
java -Xms512m programName
1 Comment
Peter Lawrey
The heap is a 1/4 of main memory by default unless you are running the 32-bit windows version. If you need to process 3 GB of files you probably need 2-4 x this much heap to load at once.