Python async subprocess pipe read nbytes

Question

After looking at the subprocess async documentation I'm left wondering how anyone would run something equivalent to await process.stdout.read(NUMBER_OF_BYTES_TO_READ). The usage of the previous code snippet is advised against right in the documentation, they suggest to use the communicate method, and from what I can tell there is no way of indicating the number of bytes that need to be read with communicate().

What am I missing? How would I tell communicate to return after reading a certain number of bytes?

Edit - I'm creating my subprocess with async pipes, I am trying to use the pipe asynchronously.

The warning is IMHO too strong, await p.stdout.read(...) is a perfectly fine way of reading the program's output. It is only if you also need to feed the program some input that a naive sequential implementation of write+read can deadlock. See this answer for a description. — user4815162342
– user4815162342, Commented Jan 10, 2020 at 22:44

Serjx86 · Accepted Answer · 2020-01-10 16:15:40Z

-1

Short answer: stdout.read is blocking.

When there is enough bytes to read, it will return. This is a very happy and unlikely occasion. More likely, there will be little or no bytes to return, so it will wait. Locking the process.

The pipe can be created to be non-blocking, but that behavior is system-specific and fickle in my experience.

The "right" way to use stdout.read is to either be ready in the reading process to be blocked on this operation, possibly indefinitely; or use an external thread to read and push data to a shared buffer. Main thread can then decide to either pull or await on the buffer, retaining control.

In practical terms, and I wrote code like this several times, there will be a listening thread attached to a pipe, reading it until close or a signal from main thread to die. Reader and Main thread will use Queue.Queue to communicate, which is trivial to use in this scenario -- it's thread safe.

So, stdout.read comes with so many caveats, that nobody in the right mind would advise anyone to use it.

answered Jan 10, 2020 at 16:15

Serjx86

5304 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ben.weller Over a year ago

Ok I guess I should have been more clear. I am using the subprocess with async stdin and stdout. I've edited the question. Similar to the example at the top of this page.

Serjx86 Over a year ago

So, the docs say the same: "This avoids deadlocks due to streams pausing reading or writing and blocking the child process." await is not magic. If underlying operation blocks, the awaited result will block too. I think what documenters intent was is to make sure people use communicate(), instead of writing their own shitty implementation of it.

Collectives™ on Stack Overflow

Python async subprocess pipe read nbytes

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related