Using threads and processes together with shared queues in Python

Question

I'm trying to use a multithreading queue and a multiprocessing queue at the same time. The threading queue will be used by 20 threads to retrieve many web pages. I then want to put the pages into a multiprocess queue so that 4 process workers can crunch the data. Below is my basic structure. My issue is that, the work queue, gives an error saying Queue is not iterable. I think the multithreading queue is overwriting the multiprocess queue but I really don't know what's wrong.

ticker_queue = Queue()

work_queue = Queue()
tickers = get_tickers()


for i in tickers:
    ticker_queue.put(i)

for i in range(20):
    t = Thread(target=network_worker, args = (ticker_queue, work_queue)).start()

for i in range(4):
    p = Process(target = worker, args = (work_queue)).start()

Here is the traceback

Traceback (most recent call last):
  File "OneDrive\Python\andys.py", line 108, in <module>
    p = Process(target = worker, args = (work_queue)).start()
  File "C:\Python27\lib\multiprocessing\process.py", line 104, in __init__
    self._args = tuple(args)
TypeError: 'Queue' object is not iterable

boil your problem down to a simple self-contained example, then post your example and the traceback. Nobody's seen professor Xavier for a while.. ;-) — thebjorn
– thebjorn, Commented Jan 24, 2015 at 18:31
I have to go to work in a few, I will post a self-contained example later tonight if no one has responded. Thanks. — nicholas.reichel
– nicholas.reichel, Commented Jan 24, 2015 at 18:35
You're missing a comma: p = Process(target = worker, args = (work_queue,)).start() — thebjorn
– thebjorn, Commented Jan 24, 2015 at 18:36
WOW! A COMMA! Thanks a bunch! It worked. Programming syntax sometimes... -.- — nicholas.reichel
– nicholas.reichel, Commented Jan 24, 2015 at 18:40

thebjorn · Accepted Answer · 2015-01-24 18:42:33Z

18

There is a comma missing in

p = Process(target = worker, args = (work_queue)).start()

since

(work_queue)

is just an expression, while what you want is a 1-element tuple:

(work_queue,)

notice the additional comma.

answered Jan 24, 2015 at 18:42

thebjorn

27.6k12 gold badges107 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using threads and processes together with shared queues in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related