Python asyncio subprocess write stdin and read stdout/stderr continuously

Question

I'm currently on a task with subprocess in python3 asyncio. My code is simply write to stdin and read stdout/stderr simultaneously:

import asyncio


async def read_stdout(stdout):
    print('read_stdout')
    while True:
        buf = await stdout.read(10)
        if not buf:
            break

        print(f'stdout: { buf }')


async def read_stderr(stderr):
    print('read_stderr')
    while True:
        buf = await stderr.read()
        if not buf:
            break

        print(f'stderr: { buf }')


async def write_stdin(stdin):
    print('write_stdin')
    for i in range(100):
        buf = f'line: { i }\n'.encode()
        print(f'stdin: { buf }')

        stdin.write(buf)
        await stdin.drain()
        await asyncio.sleep(0.5)


async def run():
    proc = await asyncio.create_subprocess_exec(
        '/usr/bin/tee',
        stdin=asyncio.subprocess.PIPE,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    await asyncio.gather(
        read_stderr(proc.stderr),
        read_stdout(proc.stdout),
        write_stdin(proc.stdin))


asyncio.run(run())

It works pretty well but I see a warning on Python3 document page:

Warning Use the communicate() method rather than process.stdin.write(), await process.stdout.read() or await process.stderr.read. This avoids deadlocks due to streams pausing reading or writing and blocking the child process.

Does that mean the above code will fall into deadlock in some scenarios? If so how to write stdin and read stdout/stderr continuously in python3 asyncio without deadlock?

Thank you very much.

communicate waits for the subprocess to terminate. If you expect to read multiple times (e.g. read something, write a reply to stdin, read again etc) then communicate simply cannot be used. The warning deals only with the simple case of one-shot reads... — Bakuriu
– Bakuriu, Commented Aug 30, 2019 at 16:38
Thank you for your reply, so for read/write again and again will be no problem? — Nguyen Binh
– Nguyen Binh, Commented Aug 31, 2019 at 2:30

user4815162342 · Accepted Answer · 2019-09-01 10:07:23Z

20

The warning was carried over from the regular subprocess module, and warns against naive code that tries to implement simple communication that appears perfectly correct, such as:

# write the request to the subprocess
await proc.stdin.write(request)
# read the response
response = await proc.stdout.readline()

This can cause a deadlock if the subprocess starts writing the response before it has read the whole request. If the response is large enough, the subprocess will block, waiting for the parent to read some of it and make room in the pipe buffer. However, the parent cannot do so because it is still writing the response and waiting for the write to complete before starting reading. So, the child waits for the parent to read (some of) its response, and the parent waits for the child to finish accepting the request. As both are waiting for the other's current operation to complete, it's a deadlock.

Your code doesn't have that issue simply because your reads and writes are executed in parallel. Since the reader never waits for the writer and vice versa, there is no opportunity for (that kind of) deadlock. If you take a look at how communicate is implemented, you will find that, barring some debug logging, it works pretty much like your code.

edited Sep 1, 2019 at 10:07

answered Aug 31, 2019 at 7:51

user4815162342

159k22 gold badges350 silver badges418 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Nguyen Binh Over a year ago

Thank you very much for a very detail answer!

hl037_ Over a year ago

It would be very nice if the python documentation was not that scary, because the same fear as the op lead me here while my code, implemented with the same logic as the op has no problem

user4815162342 Over a year ago

@hl037_ The warning could at least be less vague. It being scary is not without merit, because it really is easy to write deadllock-prone with a reasonable-seeming pattern of "I'll just write something to the child and then wait for it to respond" (unlike you and the OP, not everyone thinks to do the two in parallel). What's worst, the deadlock often doesn't occur in testing, when the payloads are smaller, but much later, and is not pleasant to debug.

hl037_ Over a year ago

Yes, At least, they could warn first about the risk, and then say "You can use communicate() to ensure there are no deadlocks"

Collectives™ on Stack Overflow

Python asyncio subprocess write stdin and read stdout/stderr continuously

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related