Using apply_async with callback function for a pool of processes

Question

I am trying to understand how multiprocess pools work. In the following programing I created a pool of 4 processes.

And I call apply_async with a callback function that should update a list called result_list

import Queue
from multiprocessing import Process
from multiprocessing import Pool

result_list = []

def foo_pool(q): #Function for each process
    print "foo_pool" 
    if(q.qsize() > 0):
        number = q.get()
    return number * 2

def log_result(result):
    # This is called whenever foo_pool(i) returns a result.
    # result_list is modified only by the main process, not the pool workers.
    result_list.append(result)

if __name__ == "__main__": 
    q = Queue.Queue()
    for i in range(4):
        q.put(i + 1) #Put 1..4 in the queue

    p = Pool(4)
    p.apply_async(foo_pool, args = (q, ), callback = log_result)

I realize I don't need to use a queue here. But I am testing this for another program which requires me to use a queue. When I run the program, the function foo_pool is not being called. The print statement print "foo_pool" does not execute. Why is this?

Sergey Gornostaev · Accepted Answer · 2016-04-28 05:00:58Z

4

Roughly speaking, apply_async only schedule async task, but not run it. You need to call p.close() and p.join() to trigger execution or r = p.apply_async() and r.get().

answered Apr 28, 2016 at 5:00

Sergey Gornostaev

7,8373 gold badges32 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

FooBar Over a year ago

But p.join() and r.get() both block until the task is done. What's the point of async-schedulign the task, if you still have to wait until it's done eventually?

Milind R Over a year ago

@FooBar The task has to be done eventually. Sergey is possibly mistaken about p.join() triggering the execution of apply_async() (it is not true as per my observations). But you can choose to call p.join only when you eventually NEED the results. Till then you can run other tasks, as opposed to waiting from the time it is called till the time it is done.

Collectives™ on Stack Overflow

Using apply_async with callback function for a pool of processes

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related