2

Theres a few dozen sqlite files i need to process and they are taking a few minutes each. Is there a way to copy or load the db into memory? IO (specifically reads as no writes are being code) is the bottleneck taking up 70% of the time.

Is there a short/quick solution i can use? Note: It will finish in <40minutes so if it isnt a quick solution it may not be worth the bother.

Improve this question

1 Answer 1

Reset to default
1

Ok, this is a "way out there" idea, and with 40 min overhead may not be worth a shot.

This doc may be of help.Basically, you copy an on-disk DB to a memory one, similar to a Python example here, then dump it back to disk. Instead of many small writes, you'll be doing a couple large ones.

Again, some testing will be needed to check your specific situation and whether it will help.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

this looks cool, but i dont know the syntax to copy it all from file to memory. I dont know how to do a insert into ThatDBOrFile from thisDBOrFile.
have you looked at the Python example I suggested. Here's a Java example. Actually, my method is not that outlandish, just looks weird.
+1. I dont have any of the functions i see in the python example such as iterdump and executescript. The java looks more similar but not the same. Its close enough.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.