Python stdin buffering issue when passing input to shell command

Question

I'm running the following script with Python 3:

import sys, subprocess

line = sys.stdin.readline()
print(f'[p] {line}', end='')
subprocess.run('read one; echo "[s] $one"', shell=True)
line = sys.stdin.readline()
print(f'[p] {line}', end='')

If I run this script normally, this happens, as I expected:

a
[p] a
b
[s] b
c
[p] c

(I've typed three lines, with a single letter on each). However, if I create a text file containing these three lines, and run it using python3 script.py < file.txt, I get the following output:

[p] a
[s]
[p] b

When I expect to get

[p] a
[s] b
[p] c

I'm fairly sure this is due to an stdin buffering issue. I've tried (to no avail) setting PYTHONUNBUFFERED, and running Python with -u. Can I disable stdin buffering? Or can I access the contents of the buffer to pass them to my shell command?

Thanks!

Arndt Jonasson · Accepted Answer · 2018-03-23 19:32:51Z

1

It is a buffering issue, and I think this statement from the documentation of the io module is relevant:

"TextIOWrapper, which extends it [talking about TextIOBase], is a buffered text interface to a buffered raw stream"

The type of sys.stdin is TextIOWrapper, so it seems the buffering can't be got rid of using just sys.stdin.

However, it seems you can do this:

import io
f = io.FileIO(sys.stdin.fileno())

and then do f.readline(), independent of whether you specify -u or not.

The question remains whether this is a good idea at all. I think an implementation which relies so critically on exactly how much is buffered is very unreliable.

answered Mar 23, 2018 at 19:32

Arndt Jonasson

8541 gold badge6 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Arndt Jonasson Over a year ago

You seem to run at least python 3.5. I tried with python 2, and there, -u with your code did make a difference.

Collectives™ on Stack Overflow

Python stdin buffering issue when passing input to shell command

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related