how to use multiprocessing/multithreading on my code?

Question

I am using the below code for a dictionary of like 100,000 keys and values...I wanted to make it more faster by doing multiprocessing/multithreading since each loop is independent of another loop. Can anyone tell me how to apply and which one (multiprocessing/multithreading) is more apt for this kind of approach

from urlparse import urlparse

ProcessAllURLs(URLs)

ProcessAllURLs(URLs)
def ProcessAllURLs(URLs):
    for eachurl in URLs:
            x=urlparse(eachurl)
            print eachurl.netloc

Thanks

Multimedia Mike · Accepted Answer · 2012-03-31 01:14:16Z

1

I would recommend Python's multiprocessing library. In particular, study the section labeled "Using a pool of workers". It should be pretty quick to rework the above code so that it uses all available cores of your system.

One tip, though: Don't print URLs from the pool workers. It is better to pass back the answer to the main process and aggregate them there for printing. Printing from different processes will result in a lot of jumbled, uncoordinated console output.

answered Mar 31, 2012 at 1:14

Multimedia Mike

13.4k5 gold badges49 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ken · Accepted Answer · 2012-03-31 02:40:33Z

1

The multiprocessing library is probably best for your example. It looks like your code could be rewritten to be:

from urlparse import urlparse

nprocs = 2 # nprocs is the number of processes to run
ParsePool = Pool(nprocs)
ParsedURLS = ParsePool.map(urlparse,URLS)

The map function is the same as the built-in map function, but runs a separate process for each function call.

See http://docs.python.org/library/multiprocessing.html for more on multiprocessing.

answered Mar 31, 2012 at 2:40

Ken

1,8881 gold badge12 silver badges10 bronze badges

Collectives™ on Stack Overflow

how to use multiprocessing/multithreading on my code?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related