Writing DB data to a binary file

Question

I'm dealing with a large scale DB that grows every day. Pulling the required data from this DB involves some joins and due to the large amount of data it takes too long. A friend suggested the following:

Once a day pull all the required data from this DB and write it to a binary file which will reside in your source-control. Next, create a dal implementation which will work against this binary file, this way things should work smoothly.

I'm not familiar with this methodology and I'm wondering - is it a good practice? What are the advantages and disadvantages of such a practice and finally is there any reference for such an implementation (currently I'm using JPA)?

Thanks in advance

James · Accepted Answer · 2012-09-10 17:37:29Z

1

A few things,

You could probably optimize how you are querying your data. See, http://java-persistence-performance.blogspot.com/2010/08/batch-fetching-optimizing-object-graph.html

If you enable a cache in JPA (or EclipseLink) then you could avoid having the query the database for the data. See, http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Basic_JPA_Development/Caching

You could archive the data in the database to avoid it growing to big. Or partition the data so only the current data is in the table.

You could use a local or in-memory database for your current data, or use a product such as Oracle TimesTen, or a caching product such as Oracle Coherence.

answered Sep 10, 2012 at 17:37

James

18.5k13 gold badges100 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

forhas Over a year ago

Thank you James, I will sure look into those!

Collectives™ on Stack Overflow

Writing DB data to a binary file

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related