Python: catch exception from subprocess while iterating through stdout

Question

I am trying to get an exception from a subprocess. I can get it if I use .communicate, but I would like to avoid using that since I am streaming output from the subprocess as it occurs, and dont want to wait until the whole subprocess is complete. Also assume the entire subprocess can take a very long time. Was wondering how I can catch an exception thrown while streaming the stdout from subprocess.

Consider the example below, so I would like to get version #1 working, version #2 kinda works, but dont want it that way.

In main.py

import subprocess


class ExtProcess():
    def __init__(self, *args):
        self.proc = subprocess.Popen(['python', *args], stdout=subprocess.PIPE)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if self.proc.returncode:
                raise Exception("error here")
            if not line:
                raise StopIteration
            return line


def run():
    ## version #1
    reader = ExtProcess("sample_extproc.py")
    for d in reader:
        print(f"got: {d}")

    ## version #2
    # proc = subprocess.Popen(['python', "sample_extproc.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # output, error = proc.communicate()
    # print("got:", output)
    # if proc.returncode:
    #     raise Exception(error)

def main():
    try:
        print("start...")
        run()
        print("complete...")
    except Exception as e:
        print(f"Package midstream error here: {str(e)}")
    finally:
        print("clean up...")


if __name__ == "__main__":
    main()

In sample_extproc.py

for x in range(10):
    print(x)
    if x == 3:
        raise RuntimeError("catch me")

I would like to get an output of something like below, from version #1:

start...
got: b'0\r\n'
got: b'1\r\n'
got: b'2\r\n'
got: b'3\r\n'
Package midstream error here: b'Traceback (most recent call last):\r\n  File "sample_extproc.py", line 4, in <module>\r\n    raise RuntimeError("catch me")\r\nRuntimeError: catch me\r\n'
clean up...

Basically it iterates through the stdout from the subprocess, then prints the exception when it occurs, then continues performing cleanup.

Programs don't return an exit status until they exit. Does your subprocess exit when it throws an exception? — Charles Duffy
– Charles Duffy, Commented Oct 14, 2020 at 17:37
...so there's very little point to trying to check returncode while you're still reading data. Sure, the process could close its stdout before it exits, or for a few tens of bytes to still be in the FIFO at exit time, but it's very unusual for that to be more than a matter of milliseconds, and even then the parent process won't know the exit status until it invoked the wait() syscall to retrieve it (when reaping the PID of the exited process from the zombie entry it leaves in the process table for the time between the exit call and its parent process reading that status). — Charles Duffy
– Charles Duffy, Commented Oct 14, 2020 at 17:38
I do feel like asking 'what is your larger goal?' might be appropriate here. I'm a fan of 'answer the question that was asked' but I can't help but feel there is another way to approach the problem (depending on what it is). — Marcel Wilson
– Marcel Wilson, Commented Oct 14, 2020 at 17:40
...if your child process exits as soon as it sees an exception, that exception won't be "in the middle", but will be at the end of the stream, so you can go straight from your for line in proc.self.stdout: loop exiting to the p.wait(), and checking p.returncode as soon as wait returns. — Charles Duffy
– Charles Duffy, Commented Oct 14, 2020 at 17:46
@CharlesDuffy Sorry just understood what you mentioned. So yea, it actually works if after I loop through the stdout, have p.wait, then check p.returncode — user1179317
– user1179317, Commented Oct 14, 2020 at 18:46

user1179317 · Accepted Answer · 2020-10-14 20:37:46Z

1

Here's the answer to my question below, really based on @CharlesDuffy's comment:

In short, make sure to have stderr=subprocess.PIPE in ExtProcess class, then the answer is in version #3, where after iterating through the stdout, we utilize .wait() and returncode to check if there was an error, if so raise an exception, grabbing the error from stderr.read() to be catched in the parent/main.

import subprocess

class ExtProcess():
    def __init__(self, *args):
        self.proc = subprocess.Popen(['python', *args], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

    def __iter__(self):
        return self

    def __next__(self):
        while True:
            line = self.proc.stdout.readline()
            if not line:
                raise StopIteration
            return line


def run():
    ## version #1
    # reader = ExtProcess("sample_extproc.py")
    # for d in reader:
    #     print(f"got: {d}")

    ## version #2
    # proc = subprocess.Popen(['python', "sample_extproc.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    # output, error = proc.communicate()
    # print("got:", output)
    # if proc.returncode:
    #     raise Exception(error)

    ## version #3
    reader = ExtProcess("sample_extproc.py")
    for d in reader:
        print(f"got: {d}")
    reader.proc.wait()
    if reader.proc.returncode:
       raise Exception(reader.proc.stderr.read())

def main():
    try:
        print("start...")
        run()
        print("complete...")
    except Exception as e:
        print(f"Package midstream error here: {str(e)}")
    finally:
        print("clean up...")


if __name__ == "__main__":
    main()

answered Oct 14, 2020 at 20:37

user1179317

2,9534 gold badges43 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Charles Duffy Over a year ago

So, the only thing that worries me reading this code is if your program tries to write more content than will fit in the pipeline to stderr before it's closed stdout, that write could block because nothing's reading it (stderr isn't just for errors -- it's also where "diagnostic content" like logs belong); so you could end up in a deadlock if the program doesn't finish writing to stdout (or doesn't attempt to close stdout) until that write to stderr completes.

user1179317 Over a year ago

@CharlesDuffy Hmm not sure if I fully understand, but is there a timeout of something I could use? I am assuming whenever I get to p.wait(), stdout has reached the StopIteration, so I shouldnt really wait too long at p.wait()

Charles Duffy Over a year ago

The potential problem is the child process deadlocking. Try using a child process that writes a few kb of data to stderr partway through its execution -- its attempt to write that data will hang unless something in the parent process is actively reading from stderr at that time; when you don't read from stderr at all until after stdout exited, that means that you can't safely run child processes that write more than a FIFO buffer's worth of data to stderr while they still have stdout open.

Charles Duffy Over a year ago

Another way is to use selectors; some of the answers in stackoverflow.com/questions/31833897/… go into it. Just reading from stderr into your parent process and storing the data for later reference is good enough.

Charles Duffy Over a year ago

...the point of using selectors to read from both stdout and stderr at the same time, or a thread reading from proc.stderr in the background, or so forth is to make sure that stderr gets consumed when the child process writes it, so the child process can't hang trying to wait for a full FIFO buffer attached to its stderr to have room for more data to be written (while the parent makes no effort to read from and thus empty that FIFO at all).

|

Collectives™ on Stack Overflow

Python: catch exception from subprocess while iterating through stdout

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related