0

In the existing project user can download report and it has 10 million records, this process gets data from database and writes to csv by using super csv java api then sends an email to user by attaching, it takes huge heap space to hold 10 million java objects and writing these records to csv files, because of this server is crashing and going down as application has many reports like this. is there any better way to handle this.? I red sxssfworkbook documentation and it says specified records count can keep in memory and remaining records will be pushed to hard disk but this is using to create excel files. is there any similar api to create csv files or sxssfworkbook can be used to create csv files.?

Improve this question
3
  • 1
    Did you try to increase the -Xmx-parameter? Can you buy more RAM? Commented Apr 8, 2018 at 13:11
  • -Xmx params are good for usual scenario's in application, only problem is with reports which has 10 millions records, infrastructure team will not agree to increase memory. Commented Apr 8, 2018 at 13:26
  • I'm not sure I understood your processes. Maybe you can compress the data? CSV-files can often be compressed dramatically. Commented Apr 8, 2018 at 16:51

1 Answer 1

Reset to default
3

There are few Java libraries for reading and writing CSV files. They typically support "streaming", so they do not have the problem of needing to hold the source data or the generated CSV in memory.

The Apache Commons CSV library would be a good place to start. Here is the User Guide. It supports various of flavors of CSV file, including the CSV formats generated by Microsoft Excel.

However, I would suggest that sending a CVS file containing 10 million records (say 1GB uncompressed data) is not going to make you popular with the people who run your users' email servers! Files that size should be made available via a web or file transfer service.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you Stephen, I will play with Apache commons csv api and I will consider to use file transfer service instead of email.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.