How does the callback function work in multiprocessing map_async?

Question

It cost me a whole night to debug my code, and I finally found this tricky problem. Please take a look at the code below.

from multiprocessing import Pool

def myfunc(x):
    return [i for i in range(x)]

pool=Pool()

A=[]
r = pool.map_async(myfunc, (1,2), callback=A.extend)
r.wait()

I thought I would get A=[0,0,1], but the output is A=[[0],[0,1]]. This does not make sense to me because if I have A=[], A.extend([0]) and A.extend([0,1]) will give me A=[0,0,1]. Probably the callback works in a different way. So my question is how to get A=[0,0,1] instead of [[0],[0,1]]?

What operating system and python version are you using?

Nafiul Islam
– Nafiul Islam

2013-10-31 05:44:04 +00:00
Commented Oct 31, 2013 at 5:44 — Nafiul Islam
– Nafiul Islam, Commented Oct 31, 2013 at 5:44

falsetru · Accepted Answer · 2013-10-31 05:55:17Z

50

Callback is called once with the result ([[0], [0, 1]]) if you use map_async.

>>> from multiprocessing import Pool
>>> def myfunc(x):
...     return [i for i in range(x)]
... 
>>> A = []
>>> def mycallback(x):
...     print('mycallback is called with {}'.format(x))
...     A.extend(x)
... 
>>> pool=Pool()
>>> r = pool.map_async(myfunc, (1,2), callback=mycallback)
>>> r.wait()
mycallback is called with [[0], [0, 1]]
>>> print(A)
[[0], [0, 1]]

Use apply_async if you want callback to be called for each time.

pool=Pool()
results = []
for x in (1,2):
    r = pool.apply_async(myfunc, (x,), callback=mycallback)
    results.append(r)
for r in results:
    r.wait()

answered Oct 31, 2013 at 5:55

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Nafiul Islam Over a year ago

This code should not work, you need to put it under a if __name__ == '__main__' statement

falsetru Over a year ago

@GamesBrainiac, (in Windows).

Nafiul Islam Over a year ago

Oh! Sorry then. You lucky mac/linux guys :P

user2727768 Over a year ago

@falsetru, it works. Thank you. Now I see a another difference between apply_async and map_async.

martineau Over a year ago

Using putting an if __name__ == '__main__' would make it work in Windows, as well as in other OSs, so I would suggest always doing it that way for improved portability.

Collectives™ on Stack Overflow

How does the callback function work in multiprocessing map_async?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related