python subprocess multiple stdin.write and stdout.read

Question

Thanks for taking time to answer the question. I am playing around with Python 3.4 and I have two simple python programs. One, a program called test.py that takes a user input and prints something.

while True:
    print("enter something...")
    x = input()
    print(x)
    time.sleep(1)

To send input to this program, I have another program that uses subprocess:

from subprocess import Popen, PIPE

cat = Popen('python test.py', shell=True, stdin=PIPE, stdout=PIPE)
cat.stdin.write("hello, world!\n")
cat.stdin.flush()
print(cat.stdout.readline())

cat.stdin.write("and another line\n")
cat.stdin.flush()
print(cat.stdout.readline())

However when I run the above program, I get an error:

enter something...

hello, world!
Traceback (most recent call last):
  File "/opt/test.py", line 9, in <module>
    x = input()
EOFError: EOF when reading a line
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

And if I replace test.py with a standard linux command like 'cat', things work as expected.

Is there any way I send multiple stdin writes and read multiple stdout back?

I think you missed the command python on Popen Popen(['python', 'test.py'], shell=True, stdin=PIPE, stdout=PIPE) — Moraes
– Moraes, Commented Oct 24, 2015 at 1:32
Typo, the program actually runs if you see that the first print works. — waka-waka-waka
– waka-waka-waka, Commented Oct 24, 2015 at 1:57
cat.stdin.write("hello, world!\n") should raise an error if you use Python 3 (universal_newlines=True is required to enable the text mode) therefore either your actual code is different or you are not using Python 3. — jfs
– jfs, Commented Oct 24, 2015 at 13:36
unrelated: you should import the module and call its functions (use multiprocessing if necessary) instead of running it as a subprocess — jfs
– jfs, Commented Oct 24, 2015 at 14:42

jfs · Accepted Answer · 2017-07-19 15:19:18Z

5

In general, you should use pexpect for interactive programs (dialog-based interactions).

Your specific issue might be caused by a python version mismatch (you think your code is executed using Python 3 while actually it might be executed using Python 2). The second issue (EOFError) is expected: either catch it in the child script or provide a signal for the child to exit (I use an empty line for that in the code example below).

Here's a Python 3 code that fails loudly on Python 2:

#!/usr/bin/env python3
import sys
from subprocess import Popen, PIPE

with Popen([sys.executable, '-u', 'test.py'], stdin=PIPE, stdout=PIPE,
           universal_newlines=True, bufsize=1) as cat:
    for input_string in ["hello, world!", "and another line", ""]:
        print(input_string, file=cat.stdin, flush=True)
        print(cat.stdout.readline(), end='')

Note:

sys.exectable is the current python executable (Python 3 in this case)
universal_newlines=True enables the text mode (otherwise, cat.stdin and cat.stdout work with bytes, not strings)
-u makes the child's output line-buffered (otherwise, you won't see anything until the child flushes its internal stdout buffer)
the with-statement closes the pipes and waits for the child process to exit.

And here's the corresponding test.py:

#!/usr/bin/env python3
import time

while True:
    x = input("enter something...")
    if not x: # exit if the input is empty
        break
    print(x)
    time.sleep(1)

Output

enter something...hello, world!
enter something...and another line
enter something...

Note: there is no new line after "enter something..."

It works but it is fragile, read Q: Why not just use a pipe (popen())? and use pexpect instead.

If the input is finite and it doesn't depend on the output then you could pass it all at once:

#!/usr/bin/env python3
import sys
from subprocess import check_output

output = check_output([sys.executable, 'test.py'],
                      input="\n".join(["hello, world!", "and another line"]),
                      universal_newlines=True)
print(output, end='')

This version requires that the child handles EOF properly:

#!/usr/bin/env python3
import time

while True:
    try:
        x = input("enter something...")
    except EOFError:
        break # no more input

    print(x)
    time.sleep(1)

The output is the same (as shown above).

edited Jul 19, 2017 at 15:19

answered Oct 24, 2015 at 14:17

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

waka-waka-waka Over a year ago

Thanks so much for the detailed explanation Sebastian! :). How do I read multiple lines? using cat.stdout.readline() will hang forever?

jfs Over a year ago

@waka-waka-waka: both code examples already read multiple lines

waka-waka-waka Over a year ago

thanks, my question was more on the lines of, I am not sure how many lines to read, when do I break? The test program may print 1 or more lines...

jfs Over a year ago

check_output() -based solution reads all output, no matter how many lines a subprocess produces.

waka-waka-waka Over a year ago

to make it more clear, assume the case where the child does not print anything to the screen, in which case the .readline() will hang forever?

|

Collectives™ on Stack Overflow

python subprocess multiple stdin.write and stdout.read

1 Answer 1

Output

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Output

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related