Node.js can't read python subprocess stdout when it read from stdin

Question

I have a node.js script which starts a python subprocess and reads its stdout. This works as long as the python process does not try to read from stdin. Then the parent process does not get anything from the child.

I have the node.js script and two python test cases here: (both examples work if you comment the lines that try to read from stdin)

First child:

import sys

print('before')

for line in sys.stdin:
    print(line)

print('after')

Second child:

import sys 

print('before')

while True:
    line = sys.stdin.readline()

    if line != '':
        print(line)
    else:
        break

print('after')

Parent:

const spawn = require('child_process').spawn;

let client = spawn('python', ['test1.py'], {cwd: '/tmp'});

client.stdout.on('data', (data) => {
  console.log(data.toString());
});

client.stderr.on('data', (data) => {
  console.log(data.toString());
});

client.on('close', () => {
  console.log('close');
});

client.on('exit', () => {
  console.log('exit');
});


client.on('disconnect', () => {
  console.log('disconnect');
})

I don't know node.js but from the python perspective a line is written but since its a pipe, not a tty, it is buffered waiting for more data. You can do print('before', flush=True) to send immediately. Then it waits for data and ... well... you need to send data. — tdelaney
– tdelaney, Commented Nov 24, 2016 at 17:12
The flush=True trick did solve the problem. If you post this as an answer I will accept it :) — Martin
– Martin, Commented Nov 24, 2016 at 17:52
It might be better to fix this on the node.js side with something like const spawn = require('pty.js').spawn; This question discusses splitting stdout/err streams stackoverflow.com/questions/15339379/…. I'm happy to supply an answer... but I'm not sure its the best answer. — tdelaney
– tdelaney, Commented Nov 24, 2016 at 17:57

tdelaney · Accepted Answer · 2016-11-24 18:05:36Z

6

A process stdout can be unbuffered, line buffered or block buffered depending on how the process was started. In particular, programs started from the console are line buffered and programs whose stdout is redirected (to a pipe or file) are block buffered. This is done to increase overall program efficiently. People want to see things right away so terminals are line buffered but other programs and files can wait and get things in bigger blocks, so they are block buffered.

You can fix the problem on the python side by forcing the data to be fluxhed on each write. You can do this with the print statement or with the sys.stdout object itself

print('line 1', flush=True)
print('line 2')
print('line 3')
sys.stdout.flush()

You can also fix it on the node.js side by emulating a terminal, basically tricking the program into thinking it is displaying to a user.

const spawn = require('pty.js').spawn;

This is more generic - you don't need the child's cooperation to make it work. But it can get complicated. Some child processes get information about the attached tty to do more complicated things like create menus or color output. But its frequently a great choice.

answered Nov 24, 2016 at 18:05

tdelaney

77.9k6 gold badges91 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Martin Over a year ago

Thanks for your answer. The pty solution works but I would like to take another look at the first idea. When I flush stdout I can receive the output from python but I have another problem: when I send text to the child process using client.stdin.write('test') this does not show up in the python program. Do I have to somehow flush stdin as well?

tdelaney Over a year ago

I think so. I don't really know node.js but if spawn sets up a pipe to the child stdin, its block buffered and has the same issues. Since you are reading lines, you also need to include a newline.

Nor.Z · Accepted Answer · 2024-03-14 19:40:54Z

Thanks to the hint of line buffer.

For my case, the solution is to add a newline at the end (+ '\n') when writing to py stdin in js. --ie: this.python.stdin.write(JSON.stringify({ word_toLemmatize }) + '\n');

python side:

try:
    for line in sys.stdin:
        # Process the data
        result = process_data(line.strip())

        # Send the result back to Node.js
        print(json.dumps(result))
        # print(json.dumps(result), flush=True)
        # print("\n", flush=True)
        sys.stdout.flush()
except EOFError:
    # End of input, exit the script
    sys.exit()

note: using this.python.stdin.write(JSON.stringify({ word_toLemmatize })); this.python.stdin.end(); could work instead for a one-time stdin write. but in my case, it leads to exit, I dont want that.
related:

node.js - Nodejs child process wont return data from python script to stdout - Stack Overflow
Nodejs child process wont return data from python script to stdout

node.js - ChildProcess: stdin not getting read by Python readline - Stack Overflow
ChildProcess: stdin not getting read by Python readline

Node.js can't read python subprocess stdout when it read from stdin - Stack Overflow
Node.js can't read python subprocess stdout when it read from stdin

Collectives™ on Stack Overflow

Node.js can't read python subprocess stdout when it read from stdin

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related