File writing in python

Question

I have a situation where my script parse approx 20000 entries and save them to db. I have used transaction which takes around 35 seconds to save and also consume high memory since until committed queries are saved in memory.

I have Found another way to write CSV then load into postgres using "copy_from" which is very fast. If anyone can suggest that if I should open file once at start then close file while loading to postgres or open file when single entry is ready to write then close.

what will be the best approach to save memory utilization?

hd1 · Accepted Answer · 2013-02-15 07:47:35Z

1

Reduce the size of your transactions?

answered Feb 15, 2013 at 7:47

hd1

34.9k5 gold badges83 silver badges95 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

sharafjaffri Over a year ago

If I commit 1000 records or 5000 records then accumulative time of 20000 records increases from single 20000 records transaction

hd1 Over a year ago

It's a trade-off between memory usage and speed ALWAYS.

sharafjaffri Over a year ago

Whats is the best size of transaction?

Collectives™ on Stack Overflow

File writing in python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related