Pipe between python scripts

Question

I have a simple script that reads values from a device and outputs them via print, and another script, which listens on stdin and interprets each number. The device outputs one number each second. Surprisingly, piping the scripts on my ubuntu box does not work. However, if the first script is made not to read from the device but generate random numbers as fast as it can, the second script successfully receives the data.

Below is a simplified example of my situation.

print.py:

#!/usr/bin/env python2
import time
import sys

while True:
    time.sleep(1)  # without this everything works
    print "42"
    sys.stdout.flush()

read.py:

#!/usr/bin/env python2
import sys

while True:
    for str in sys.stdin:
        print str

Command line invocation:

vorac@laptop:~/test$ ./print.py | ./read.py

Here is the end result. The first script reads from the device and the second graphs the data in two separate time frames (what is shown are random numbers).

I just added newin = os.fdopen(sys.stdin.fileno(), 'r', 0); sys.stdin = newin to read.py and nothing has changed. I would strongly prefer to resolve this programatically and keep the invocation of the scripts simple. — Vorac
– Vorac, Commented Nov 9, 2015 at 12:07
@MKesper, this is a program for reading, logging and presentation of temperature data. Being able to use the modules independently seems like a good feature - you could just listen to the device, or just plot some random data, or run the interruptable logger on some other data. — Vorac
– Vorac, Commented Nov 9, 2015 at 13:09

David Z · Accepted Answer · 2015-11-09 12:27:21Z

Ah, now that is a tricky problem. It happens because the iterator method for sys.stdin (which is xreadlines()) is buffered. In other words, when your loop implicitly calls next(sys.stdin) to get the next line of input, Python tries to read from the real under-the-hood standard input stream until its internal buffer is full, and only once the buffer is full does it proceed through the body of the loop. The buffer size is 8 kilobytes, so this takes a while.

You can see this by decreasing the time delay in the sleep() call to 0.001 or some such value, depending on the capabilities of your system. If you hit the time just right, you'll see nothing for a few seconds, and then a whole block of 42s come out all at once.

To fix it, use sys.stdin.readline(), which is unbuffered.

while True:
    line = sys.stdin.readline()
    print line

You might also want to strip off the trailing newline before printing it, otherwise you'll get double line breaks. Use line.rstrip('\n'), or just print line, to suppress the extra newline that gets printed.

Amey Jadiye · Accepted Answer · 2015-11-09 13:15:46Z

2

I changed your read.py and it worked for me :), you forget to .readline() from stdin.

import sys
while True:
    line = sys.stdin.readline()
    if line:
        print line.strip()
    else:
        continue

Output is :

$ python write.py | python read.py 
42
42
42
42
42
42
42

edited Nov 9, 2015 at 13:15

answered Nov 9, 2015 at 12:09

Amey Jadiye

3,1643 gold badges27 silver badges40 bronze badges

1 Comment

Amey Jadiye Over a year ago

I think readLine() is important, time is useless, i was checking something else here ... corrected.

Collectives™ on Stack Overflow

Pipe between python scripts

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related