Simultaneous downloading a file and inserting to mysql database with python

Question

I'm downloading relatively large files (10mb each) with urllib2, and then loading it as a json file and inserting data into a mysql database, and then repeating the process in an infinite loop. The downloading takes a minute or so, and then it loads everything into mysql. Is there a way to create a thread that does the downloading while the main thread inserts into mysql using python?

my pseudocode:

while 1:
 download file with urllib2
 decode as json file
 extract data I want
 do some computations on data
 insert data into mysql

Thank you so much!

Twisted (as said by Ignacio) or Tornado, they are both Asynchronous. — Fabio Buda
– Fabio Buda, Commented Jan 19, 2012 at 6:29

philofinfinitejest · Accepted Answer · 2012-01-19 06:23:25Z

2

What you could do is use threads and queues. A file IO thread would read and process the file and then insert the result into a queue, where a database IO thread would then discover the result and do the work. Rather than code up an example, I'll direct you here: http://www.ibm.com/developerworks/aix/library/au-threadingpython/

Alternately you could use the python select module to manage multiple file read operations and handle them one by one as they complete: http://docs.python.org/library/select.html

answered Jan 19, 2012 at 6:23

philofinfinitejest

4,0772 gold badges26 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Fabio Buda Over a year ago

Maybe multiprocessing (or process pool) colud be safer.

philofinfinitejest Over a year ago

A good point in the case where multiple processors would be useful: a CPU bound operation. In an IO bound situation the extra processors aren't the worry - getting the IO not to block is.

jh314 Over a year ago

Awesome, just what I was looking for. So I just have to modify the "run" function in the given example with my own code?

philofinfinitejest Over a year ago

Right -- the python thread's start method hands off to the run method where you define the work that the thread will do. So you would likely have one thread class whose run method downloads and processes the files, another thread class whose run method does database inserts. And as the example demonstrates it is easy to set up a thread pool so you could be downloading/inserting multiple files at the same time.

Collectives™ on Stack Overflow

Simultaneous downloading a file and inserting to mysql database with python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related