3

How can I access the stdout of child processes prior to sending them to main process? I am using multiprocessing.Pool module to generate child process pools.

3
  • maybe this could help you: stackoverflow.com/questions/7714868/… Commented Oct 17, 2013 at 7:41
  • @Udy the question you reference seems like a windows specific stdout redirection issue. Also, they're trying to redirect the stdout to a file. I on the other hand want to just modify the child process's stdout. Commented Oct 17, 2013 at 8:00
  • true. but this will defantly help: stackoverflow.com/questions/4227808/… Commented Oct 17, 2013 at 8:14

1 Answer 1

9

The main process and the children all share the same standard input and standard output file descriptors. They have no control over what the other writes to them. The only thing you can do is replace stdin and stdout in the children with something else that the main process can control. As an example you could subclass a dummy file object like StringIO and redirect the data that the children write to this object to the parent via a Queue:

import sys
from multiprocessing import Queue, Pool, current_process
from StringIO import StringIO

class MyStringIO(StringIO):
    def __init__(self, queue, *args, **kwargs):
        StringIO.__init__(self, *args, **kwargs)
        self.queue = queue
    def flush(self):
        self.queue.put((current_process().name, self.getvalue()))
        self.truncate(0)

def initializer(queue):
     sys.stderr = sys.stdout = MyStringIO(queue)

def task(num):
     print num
     sys.stdout.flush()
     return num ** 2

q = Queue()
pool = Pool(3, initializer, [q])

for _ in pool.map(task, range(5)):
    proc, out = q.get()
    print proc, "got", out

This should print something like this:

PoolWorker-1 got 0
PoolWorker-1 got 3
PoolWorker-1 got 4
PoolWorker-2 got 1
PoolWorker-3 got 2

Don't forget to call sys.{stdout,stderr}.flush() at the end of task otherwise, nothing will be written to the queue.

Sign up to request clarification or add additional context in comments.

1 Comment

the children's output is redirected to the MyStringIO object which then sends the data to the main process via the queue. The main process then does all the processing it needs and outputs it to sys.stdout (which, as I explain in my updated answer, is shared by all processes attached to the terminal)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.