How does one create custom output stream for subprocess.call

Question

I am trying to get realtime output of a subprocess.call by defining my own output stream but it doesn't seem to work.

Reason: I want to run a subprocess and get output of that call to both stdout(in realtime so i can look at the script and see current progress) as well as logging it to a file

subprocess.py:

import time

while True:
    print("Things")
    time.sleep(1)

mainprocess.py

import subprocess
import io

class CustomIO(io.IOBase):
    def write(self, str):
        print("CustomIO: %s"%str)
        # logging to be implemented here

customio = CustomIO()
subprocess.call(["python3", "print_process.py"], stdout=customio)

But when i run this code i get this error message:

Traceback (most recent call last):                                   
    File "call_test.py", line 9, in <module>                           
        subprocess.call(["python3", "print_process.py"], stdout=customio)
    File "/usr/lib/python3.4/subprocess.py", line 537, in call         
        with Popen(*popenargs, **kwargs) as p:                           
    File "/usr/lib/python3.4/subprocess.py", line 823, in __init__     
        errread, errwrite) = self._get_handles(stdin, stdout, stderr)    
    File "/usr/lib/python3.4/subprocess.py", line 1302, in _get_handles
        c2pwrite = stdout.fileno()                                       
io.UnsupportedOperation: fileno

So, anyone have any clue if this is possible?

Am i inheriting the wrong baseclass?

Am i not overloading the proper methods?

Or am i completely off the rails and should be going about this in a completely different way?

subprocess expect a real file object with real os file descriptor, try PIPE — georgexsh
– georgexsh, Commented Sep 13, 2017 at 15:36

Phydeaux · Accepted Answer · 2017-09-14 09:18:38Z

4

If you want to process the output of a subprocess, you need to pass stdout=subprocess.PIPE. However, call() and run() will both wait until the process is finished before making it available, so you cannot handle it in real time using these functions.

You need to use subprocess.Popen:

import subprocess as sp

def handle_output(output_line):
    ...

my_process = sp.Popen(["python3", "print_process.py"],
                      stdout=sp.PIPE,
                      universal_newlines=True) # changes stdout from bytes to text

for line in my_process.stdout:
    handle_output(line)

my_process.wait()

Update: Make sure to flush the output buffer in your child process:

while True:
    print("Things", flush=True)
    time.sleep(1)

edited Sep 14, 2017 at 9:18

answered Sep 13, 2017 at 15:35

Phydeaux

2,8753 gold badges20 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Bobbzorzen Over a year ago

This does not seem to handle output in realtime either? when i run this it waits for subprocess to be completed before printing anything

Phydeaux Over a year ago

my apologies - for some reason it worked when I tested with ["yes"]. See update.

Bobbzorzen Over a year ago

Nice, is there any way to do this without adding a flush in the child process. In my real case it's not a python script but instead a program of which i have no controll over.

Phydeaux Over a year ago

I think there are a couple of ways to approach this, and it's probably worth its own question.

Bobbzorzen Over a year ago

ok, thank you. will accept this answer then since it solved the provided problem

GIZ · Accepted Answer · 2017-09-13 16:18:51Z

You need to specify and open stream with a file descriptor. fileno isn't implemented for io.IOBase because this is just an in-memory stream:

Frequently Used Arguments

stdin, stdout and stderr specify the executed program’s standard input, standard output and standard error file handles, respectively. Valid values are PIPE, DEVNULL, an existing file descriptor (a positive integer), an existing file object, and None. PIPE indicates that a new pipe to the child should be created. DEVNULL indicates that the special file os.devnull will be used. With the default settings of None, no redirection will occur;

So you might use sockets, pipes, and open files as stdout, the file descriptor is passed to the child process as it's stdout. I didn't use sockets with subprocess.Popen though, but I expect them to work, I believe what matters here is the file descriptor to the child, not what type of object the file descriptor points to.

Collectives™ on Stack Overflow

How does one create custom output stream for subprocess.call

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related