Python: program hangs when trying to read from subprocess stdout while it is running

Question

I am trying to communicate with a c++ script (let's call it script A) using the python subprocess module. Script A is running alongside the python program and is constantly interacted with. My goal is to send script A input commands, and capture the outputs that are being printed to STDOUT afterwards by script A. I'm working on windows 10.

Here is a snippet describing the logic:

proc = subprocess.Popen([".\\build\\bin.exe"], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
terminate = False

while not terminate:
    command = input("Enter your command here: ")
    if command == "q":
        terminate = True
    else:
        proc.stdin.write(command.encode()) # send input to script A
        output = proc.stdout.readline().decode() # problematic line, trying to capture output from script A
        print(f"Output is: {output}")

The problem is that while script A is writing output to STDOUT after each command like I expect it to, the python script hangs when it reaches the line highlighted above. I tried to capture the output using proc.stdout.read(1) with bufsize=0 on the call to Popen and for line in iter(proc.stdout.readlines()) and some other ways but the problem persists.

Would appreciate any help on this because nothing I tried is working for me. Thanks in advance!

Does this answer your question? Python subprocess & stdout - program deadlocks — wovano
– wovano, Commented Mar 8, 2022 at 13:32

wovano · Accepted Answer · 2022-03-08 20:55:24Z

1

You already suggested to use bufsize=0, which seems the right solution. However, this only affects buffering at the Python side. If the executable you are calling uses buffered input or output, I don't think there's anything you can do about it (as also mentioned here].

If both programs are under your own control, then you can easily make this work. Here is an example. For simplicity I created two Python scripts that interact with each other in a similar way you are doing. Note that this doesn't differ very much from the situation with a C++ application, since in both cases an executable is started as subprocess.

File pong.py (simple demo application that reads input and responds to it - similar to your "script A"):

while True:
    try:
        line = input()
    except EOFError:
        print('EOF')
        break
    if line == 'ping':
        print('pong')
    elif line == 'PING':
        print('PONG')
    elif line in ['exit', 'EXIT', 'quit', 'QUIT']:
        break
    else:
        print('what?')
print('BYE!')

File main.py (the main program that communicates with pong.py):

import subprocess


class Test:

    def __init__(self):
        self.proc = subprocess.Popen(['python.exe', 'pong.py'], bufsize=0, encoding='ascii',
                                     stdout=subprocess.PIPE, stdin=subprocess.PIPE)

    def talk(self, tx):
        print('TX: ' + tx)
        self.proc.stdin.write(tx + '\n')
        rx = self.proc.stdout.readline().rstrip('\r\n')
        print('RX: ' + rx)


def main():
    test = Test()
    test.talk('ping')
    test.talk('test')
    test.talk('PING')
    test.talk('exit')


if __name__ == '__main__':
    main()

Output of python main.py:

TX: ping
RX: pong
TX: test
RX: what?
TX: PING
RX: PONG
TX: exit
RX: BYE!

Of course there are other solutions as well. For example, you might use a socket to communicate between the two applications. However, this is only applicable if you can modify both application (e.g. if you are developing both applications), not if the executable you are calling is a third-party application.

edited Mar 8, 2022 at 20:55

answered Mar 8, 2022 at 20:42

wovano

5,1915 gold badges33 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Olvin Roght Over a year ago

Also you could apply print() to send text to process stdin directly, it will add newline. print(tx, file=self.proc.stdin).

Olvin Roght Over a year ago

And you can omit arguments from .rstrip() it will clean all whitespaces by default

wovano Over a year ago

@OlvinRoght, thanks for the good suggestions. The print(tx, file=..) is a matter of preference and I personally find the current solution more clear. And I deliberately specified the line endings in rstrip() because I don't know if the application of the OP might return other whitespace that might be important. But if not, then it's definitely cleaner to use .rstrip() without arguments (or even .strip()). I'll leave that to the OP (or other users) to decide for themselves :-)

N0I0C0K · Accepted Answer · 2022-09-11 02:32:46Z

-1

first, buffersize=0, which is the right solution. However, this is not enough.

In your executeable program, you should set the stdout buffersize to 0. or flush in time.

In C/C++ program, you can add

setbuf(stdout, nullptr);

to your source code.

answered Sep 11, 2022 at 2:32

N0I0C0K

473 bronze badges

Collectives™ on Stack Overflow

Python: program hangs when trying to read from subprocess stdout while it is running

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related