I want to execute some processes in parallel and wait until they finish. So I wrote this code:
pool = mp.Pool(5)
for a in table:
pool.apply(func, args = (some_args))
pool.close()
pool.join()
Will I get 5 processes executing func in parallel here?
Or the only option is apply_async?
The docs are quite clear on this: each call to apply blocks until the result is ready. Use apply_async.
func either via the callback or by calling get on the object returned by apply_async. In your example you don't use the return value at all. In that case you could use apply_async without further changes.mp.map instead.map(...) waits for the result, equivalent to map_async(...).get(). Unlike apply, map is able to initiate parallel work when it calls the function multiple times (for each item in the iterable argument).Another solution is to use Pool.imap_unordered()
The following code starts a pool of 5 workers. It then sends three jobs to the pool. The first one is num=1, second num=2, etc. The function imap_unordered means that when the first result appears, from any worker, return it for further processing. In this case, the loop prints results as they appear, which isn't in any specific order.
import multiprocessing
def calc(num):
return num*2
pool = multiprocessing.Pool(5)
for output in pool.imap_unordered(calc, [1,2,3]):
print 'output:',output
