How to implement a queue of worker threads processing URLs

Question

I am a new to python and having some issues with threading and queues.

Basically i have a file of URL's, and would like to create a queue of 10 workers which sequentially work down through the file and perform a urllib2 request.

I just cant get my head around it! Could anyone help please with some simple links or code?

Thanks, Matt

i think you need to show first what you already have.

aschmid00
– aschmid00

2012-04-23 20:01:14 +00:00
Commented Apr 23, 2012 at 20:01 — aschmid00
– aschmid00, Commented Apr 23, 2012 at 20:01

dkamins · Accepted Answer · 2012-04-23 21:23:20Z

5

The built in "Queue" module would make a good underlying queue for your app. It handles all multi-threaded locking semantics for you.

http://docs.python.org/library/queue.html

You can load your file, populate your Queue, spin up your worker threads, and have them pull from the Queue and do their work.

answered Apr 23, 2012 at 21:23

dkamins

22.1k7 gold badges58 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matt Storey Over a year ago

I used this link and example to help build out what i wanted in the end. Thank you.

Tom · Accepted Answer · 2012-04-23 20:01:01Z

0

This python threadpool module is very helpful for what you describe. The author has a few examples for how to use it on the main page and in the docs.

http://www.chrisarndt.de/projects/threadpool/

answered Apr 23, 2012 at 20:01

Tom

19.4k16 gold badges78 silver badges83 bronze badges

Collectives™ on Stack Overflow

How to implement a queue of worker threads processing URLs

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related