Python multiprocessing Queue failure

Question

I create 100 child processes

proc_list = [
    Process(target = simulator, args=(result_queue,))
    for i in xrange(100)]

and start them

for proc in proc_list: proc.start()

Each process puts into the result_queue (instance of multiprocessing.Queue) 10000 tuples after doing some processing.

def simulate(alg_instance, image_ids, gamma, results,
                     simulations, sim_semaphore):
  (rs, qs, t_us) =  alg_instance.simulate_multiple(image_ids, gamma,
                                             simulations)
  all_tuples = zip(rs, qs, t_us)
  for result in all_tuples:
    results.put(result)
  sim_semaphore.release()

I should be (?) getting 1000000 tuples at the queue, but after various runs I get these (sample) sizes: 14912 19563 12952 13524 7487 18350 15986 11928 14281 14282 7317

Any suggestions?

Are you joining each of the processes to make sure you are waiting for all of them to complete? — K. Brafford
– K. Brafford, Commented Jul 11, 2012 at 23:20
Naming a variable tuple is a very bad idea. tuple is a basic type and you shouldn't replace it in your namespace with something else. — K. Brafford
– K. Brafford, Commented Jul 11, 2012 at 23:23
@K.Brafford I haven't really named them tuple -- just wrote the sketch of the function. I've updated the function code. — user1451817
– user1451817, Commented Jul 11, 2012 at 23:25

bukzor · Accepted Answer · 2012-07-11 23:33:07Z

23

My solution to multiprocessing issues is almost always to use the Manager objects. While the exposed interface is the same, the underlying implementation is much simpler and has less bugs.

from multiprocessing import Manager
manager = Manager()
result_queue = manager.Queue()

Try it out and see if it doesn't fix your issues.

answered Jul 11, 2012 at 23:33

bukzor

38.8k13 gold badges85 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Hrvoje Špoljar Over a year ago

I started losing my nerves with multiprocessing.Queue, envisioned I must have made some mistake but no change really, just swapped it with manager.Queue and it started working flawlessly :\ -- thanks a bunch for advice

trading4living · Accepted Answer · 2016-09-25 00:46:39Z

6

The multiprocessing.Queue is said to be thread-safe in its documentations. But when you are doing inter-process communications with Queue, it should be used with multiprocessing.Manager().Queue()

answered Sep 25, 2016 at 0:46

trading4living

1291 silver badge5 bronze badges

Comments

max · Accepted Answer · 2017-03-13 06:36:33Z

2

There's no evidence from the OP post that multiprocessing.Queue does not work. The code posted by the OP is not at all sufficient to understand what's going on: do they join all the processes? do they correctly pass the queue to the child processes (has to be as a parameter if it's on Windows)? do their child processes verify that they actually got 10000 tuples? etc.

There's a chance that the OP is really encountering a hard-to-reproduce bug in mp.Queue, but given the amount of testing CPython has gone through, and the fact that I just ran 100 processes x 10000 results without any trouble, I suspect the OP actually had some problem in their own code.

Yes, Manager().Queue() mentioned in other answers is a perfectly fine way to share data, but there's no reason to avoid multiprocessing.Queue() based on unconfirmed reports that "something is wrong with it".

edited Mar 13, 2017 at 6:36

answered Mar 12, 2017 at 18:13

max

52.7k60 gold badges224 silver badges379 bronze badges

2 Comments

DrM Over a year ago

Actually, I have the same problem as the OP. See this stackoverflow.com/q/78925731/4867193

DrM Over a year ago

And see the comment from Hrvoje following the answer from Bukor. This is a very common and very well known problem.

Collectives™ on Stack Overflow

Python multiprocessing Queue failure

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related