Issue with Pool and Queue of multiprocessing module in Python

Question

I am new to multiprocessing of Python, and I wrote the tiny script below:

import multiprocessing
import os

def task(queue):
    print(100)

def run(pool):
    queue = multiprocessing.Queue()
    for i in range(os.cpu_count()):
        pool.apply_async(task, args=(queue, ))

if __name__ == '__main__':
    multiprocessing.freeze_support()
    pool = multiprocessing.Pool()
    run(pool)
    pool.close()
    pool.join()

I am wondering why the task() method is not executed and there is no output after running this script. Could anyone help me?

Tim Peters · Accepted Answer · 2017-04-23 17:43:06Z

It is running, but it's dying with an error outside the main thread, and so you don't see the error. For that reason, it's always good to .get() the result of an async call, even if you don't care about the result: the .get() will raise the error that's otherwise invisible.

For example, change your loop like so:

tasks = []
for i in range(os.cpu_count()):
    tasks.append(pool.apply_async(task, args=(queue,)))
for t in tasks:
    t.get()

Then the new t.get() will blow up, ending with:

RuntimeError: Queue objects should only be shared between processes through inheritance

In short, passing Queue objects to Pool methods isn't supported.

But you can pass them to multiprocessing.Process(), or to a Pool initialization function. For example, here's a way to do the latter:

import multiprocessing
import os

def pool_init(q):
    global queue # make queue global in workers
    queue = q

def task():
    # can use `queue` here if you like
    print(100)

def run(pool):
    tasks = []
    for i in range(os.cpu_count()):
        tasks.append(pool.apply_async(task))
    for t in tasks:
        t.get()

if __name__ == '__main__':
    queue = multiprocessing.Queue()
    pool = multiprocessing.Pool(initializer=pool_init, initargs=(queue,))
    run(pool)
    pool.close()
    pool.join()

On Linux-y systems, you can - as the original error message suggested - use process inheritance instead (but that's not possible on Windows).

Collectives™ on Stack Overflow

Issue with Pool and Queue of multiprocessing module in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related