Python Multiprocessing apply_async Only Using One Process

Question

This is my code:

import multiprocessing
import time
import os

def worker():
    print str(os.getpid()) + " is going to sleep..."
    time.sleep(1)
    print str(os.getpid()) + " woke up!"
    return

if __name__ == "__main__":
    pool = multiprocessing.Pool(processes = 8)
    for _ in range(5):
        pool.apply_async(worker())

And my output:

23173 is going to sleep...
23173 woke up!
23173 is going to sleep...
23173 woke up!
23173 is going to sleep...
23173 woke up!
23173 is going to sleep...
23173 woke up!
23173 is going to sleep...
23173 woke up!

This output appears sequentially and obviously all the outputs have the same id. What I'm expecting to see is 5 separate processes alerting they that are going to sleep and then waking up at similar times.

Why is this not happening? I have done a lot of research on stackoverflow and other sites and it seems like everyone is assured that this should have the "worker" callable applied across multiple processes, but it appears not to be.

should be: apply_asynch(worker) without the ().

Bakuriu
– Bakuriu

2016-07-28 19:08:28 +00:00
Commented Jul 28, 2016 at 19:08 — Bakuriu
– Bakuriu, Commented Jul 28, 2016 at 19:08

Bakuriu · Accepted Answer · 2016-07-28 19:19:42Z

You are calling the worker function, so the current process executes that code and then the result of worker (i.e. None) is passed to apply_asynch which does nothing.

Change it to pool.apply_asynch(worker) in this way the function worker is passed to the subprocess which executes it.

To be dead clear, the problem is the difference between this:

>>> def worker():
...     print('Hi there!')
... 
>>> def f(func):
...     print('Received', func)
...     func()
... 
>>> f(worker())
Hi there!
Received None
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in f
TypeError: 'NoneType' object is not callable

And this:

>>> f(worker)
Received <function worker at 0x7f5cd546f048>
Hi there!

Note that the former has the wrong order of lines and produces a TypeError. Unfortunately Pool.apply_asynch silences exceptions by default. You can handle errors passing in extra arguments though.

Collectives™ on Stack Overflow

Python Multiprocessing apply_async Only Using One Process

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related