Python subprocess stdout doesn't capture input prompt

Question

I'm using subprocess to spawn a conda create command and capture the resulting stdout for later use. I also immediately print the stdout to the console so the user can still see the progress of the subprocess:

p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
    sys.stdout.write(line.decode(sys.stdout.encoding))

This works fine until half way through the execution when the conda create requires user input: it prompts Proceed (n/y)? and waits for the user to input an option. However, the code above doesn't print the prompt and instead just waits for input "on a blank screen". Once an input is received the prompt is printed afterwards and then execution continues as expected.

I assume this is because the input somehow blocks the prompt being written to stdout and so the readline doesn't receive new data until after the input block has been lifted.

Is there a way I can ensure the input prompt is printed before the subprocess waits for user input? Note I'm running on Windows.

Note that readline actually looks for an entire line, up to the end-of-line character \n. A Proceed (n/y)? y prompt does not output an entire line but expects input (the y) inline. — MisterMiyagi
– MisterMiyagi, Commented Jul 28, 2020 at 8:27
You raise a good point. Is it likely a simple fix using read instead? — Mrfence
– Mrfence, Commented Jul 28, 2020 at 11:58

Mrfence · Accepted Answer · 2020-07-28 15:34:12Z

5

Although I'm sure pexpect would have worked in this case, I decided it would be overkill. Instead I used MisterMiyagi's insight and replaced readline with read.

The final code is as so:

p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while p.poll() is None:
    sys.stdout.write(p.stdout.read(1).decode(sys.stdout.encoding))
    sys.stdout.flush()

Note the read(1) as just using read() would block the while loop until an EOF is found. Given no EOF is found before the input prompt, nothing will be printed at all! In addition, flush is called which ensures the text written to sys.stdout is actually visible on screen.

answered Jul 28, 2020 at 15:34

Mrfence

3702 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Thomas Aumaitre Over a year ago

proc.stdout.isatty() gives False, and proc.stdout.writable() as well. Why is it that much complicated ? proc.stdout.read(1) get stuck in a infinite loop.

Rob Raymond · Accepted Answer · 2020-07-28 08:26:42Z

0

For this use case I recommend using pexpect. stdin != stdout

Example use case where it conditionally sends to stdin on prompts on stdout

def expectgit(alog):
    TMPLOG = "/tmp/pexpect.log"
    cmd = f'''
ssh -T [email protected] ;\
echo "alldone" ;
'''
    with open(TMPLOG, "w") as log:
        ch = pexpect.spawn(f"/bin/bash -c \"{cmd}\"", encoding='utf-8', logfile=log)
        while True:
            i = ch.expect(["successfully authenticated", "Are you sure you want to continue connecting"])
            if i == 0:
                alog.info("Git all good")
                break
            elif i == 1:
                alog.info("Fingerprint verification")
                ch.send("yes\r")
        ch.expect("alldone")
        i = ch.expect([pexpect.EOF], timeout=5)
        ch.close()
        alog.info("expect done - EOF")

    with open(TMPLOG) as log:
        for l in log.readlines():
            alog.debug(l.strip())

answered Jul 28, 2020 at 8:26

Rob Raymond

31.5k3 gold badges19 silver badges34 bronze badges

1 Comment

Robatron Over a year ago

This works pretty slick for getting the output to the screen. How to accomplish capturing the user response for sending back to the subprocess...?

Collectives™ on Stack Overflow

Python subprocess stdout doesn't capture input prompt

2 Answers 2

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related