How can I stream input into a python Popen subprocess and stream output from it?

Question

I'm attempting to basically create a shell in python using subprocess.Popen. As boilerplate code for testing purposes, this is what I have:

if __name__ == '__main__':
    ps = subprocess.Popen(
        input('command? '),
        shell=True,
        stdin=subprocess.PIPE,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        text=True
    )

    print(ps.stdout.read())
    time.sleep(1) # provide enough time to register completion if the command is a one-off, like dir or echo
    while ps.poll() == None: # ps.poll() returns None if still running
        try:
            # method one
            ps.stdin.write(input())
            ps.communicate()
            print(ps.stdout.read())

            # method two
            ps.stdin.write(input())
            print(ps.communicate()[0])

            # method three
            print(ps.communicate(input())[0])
        except:
            continue

    print('Process Finished')

Each method is a different (failed) attempt.

For commands like python which should open the python CLI interpreter, this completely fails. However, for one-off commands like dir or ls or even running python -c "<some python>" it works just fine.

CLI log:

C:\Users\nj2u2\Desktop\test>python test.py
command? echo hello
hello

Process Finished

C:\Users\nj2u2\Desktop\test>python test.py
command? dir
 Volume in drive C has no label.
 Volume Serial Number is D6B7-6B8D

 Directory of C:\Users\nj2u2\Desktop\test

07/03/2020  12:26 AM    <DIR>          .
07/03/2020  12:26 AM    <DIR>          ..
07/03/2020  08:20 PM             6,811 subprocess_plus.py
07/04/2020  12:55 PM               580 test.py
07/03/2020  08:25 PM    <DIR>          __pycache__
               2 File(s)          7,391 bytes
               3 Dir(s)  1,487,446,302,720 bytes free

Process Finished

C:\Users\nj2u2\Desktop\test>python test.py
command? python

After that last command, python, it just hangs at print(ps.stdout.read()).

I'd like to know why it's hanging, and how I can fix it.

If your script works with some commands and not others, perhaps the issue is not related to your script but to how those "failing" commands are implemented. Have you tried simply opening the python CLI as a subprocess, and sending it a short, hard-coded command? Does that work? If it doesn't then your script is not going to work, and there is nothing you can do about it. Programs are not required to support the customary stdin/stdout protocol, so expect that some of them will not work with your approach. — Paul Cornelius
– Paul Cornelius, Commented Jul 4, 2020 at 3:01

Serge Ballesta · Accepted Answer · 2020-07-15 09:35:32Z

The communicate method can only be used for non interactive commands because it sends a one shot input and then waits for the sub process to terminate. That is the reason that it can only work with what you call one-off commands.

But beware, pipes do not support the terminal discipline in which a read call automatically flushes the output, and where a new line causes a read to return immediately. So nothing guarantees that you can use that to provide interactive IO to the python CLI interpretor. Pipes are just streams, and can be buffered so you can just be sure that all the input will be delivered, and that when the output channels will be closed (i.e. when the sub-process command will terminate) you will receive all the output. The way input and output are intermixed depends on the way the sub-process command is programmed. If it willingly flushes its output before reading, and you do the same in your code, then interactive processing is possible, but if it does not, there is no way to expect to receive something after each end of line.

Collectives™ on Stack Overflow

How can I stream input into a python Popen subprocess and stream output from it?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related