Python multiprocessing Queue memory management

Question

Say I have the Main processes and 2 additional processes A and B. In this program A is supposed to send data to B. If we have some code like this:

from multiprocessing import Process, Queue

def process_a(iterable, q):

    for x in iterable:
        q.put(x)

def process_b(q):

    while some_condition():
        x = q.get()


iterable = some_iterable()
q = Queue()

pa = Process(target=process_a, args=(iterable, q))
pb = Process(target=process_b, args=(q,))

pa.start()
pb.start()

pa.join()
pb.join()

given that the Queue q was created in the Main process, does the data flow like this?

A => Main => B

If so, is there a way to have a Queue initialized on B and passed to A such that data goes directly from A to B skipping Main?

abarnert · Accepted Answer · 2018-09-12 02:37:15Z

7

given that the Queue q was created in the Main process, does the data flow like this?

A => Main => B

No. As explained in the docs, a Queue is just an auto-synchronizing wrapper around a Pipe. When you pass a Queue to a child, you're just passing that Pipe and some locks.

And the Pipe is just a wrapper around an operating system pipe. When you pass a Pipe to a child, you're just passing the pipe's file descriptor/handle.

Ignoring the locks, process A is basically just writing to a pipe, and process B is just reading from it.

The locks do make things a bit more complicated (and may also mean that process A spins up a hidden background thread), but they still don't involve the main process at all.

Unless the main process calls a method on a queue, it has nothing to do with that queue at all.

answered Sep 12, 2018 at 2:37

abarnert

368k54 gold badges626 silver badges691 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python multiprocessing Queue memory management

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related