How can I send a byte-object over a pipe in python?

Question

I am trying to send a numpy array over stdout from one process to another.

Sending it over the pipe requires me to convert it to a string. On the other side, I receive a byte object. This byte object now has the original byte string encapsulated.

I find it now impossible to restore the original byte object as a byte object. If I decode the byte object, I receive a string which is incompatible with all the restoring functions I tried (np.frombuffer, pickle.loads).

server.py

import subprocess
import numpy as np
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
    tmp = p.stdout.readline()    
    # doesn't fail but wrong size
    array = np.frombuffer(tmp, dtype=np.uint8)
    tmp = bytes.decode(tmp)
    # fails because byte object is necessary
    array = np.frombuffer(tmp, dtype=np.uint8)
    array = array.reshape([1, 3, 5, 5, 1])
    print(array.shape)

writer.py

import numpy as np
import sys
while 1:
    array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
    string = array.tobytes()
    sys.stdout.write(str(string))
    sys.stdout.flush()

Is there anyway to convert the string into a byteobject, without encoding it? How else could this work? I want to use pipes instead of shared memory as proposed in some other solutions to keep it more simple. Furthermore, I need it to be parallel but blocking, so Pipes seemed ideal to me.

Thanks

how about you first send the size of the object to come, then read that size from the buffer in the server? — Netwave
– Netwave, Commented Mar 6, 2017 at 8:18
@DanielSanchez I don't quite understand, there's no buffer on the server side? There's stdout from which I can read out everything that was sent. But reading the whole string is not the problem. I want to restore the original object. — Kilsen
– Kilsen, Commented Mar 6, 2017 at 8:23
@Kilsen have you tried in the writer sys.stdout.write(string) without converting to str? — MB-F
– MB-F, Commented Mar 6, 2017 at 8:25
@kazemakase Yes, this isn't possible because stdout.write requires a string — Kilsen
– Kilsen, Commented Mar 6, 2017 at 8:27
@Kilsen sys.stdout.write(b'abc') works for me. Why does it not work for you? — MB-F
– MB-F, Commented Mar 6, 2017 at 8:34

ymonad · Accepted Answer · 2017-03-06 08:46:30Z

3

you can use pickle to marshal the data, and sys.stdout.buffer instead of sys.stdout to write bytes to stdout.

Reference: sys.stdout

server.py:

import subprocess
import numpy as np
import pickle
p = subprocess.Popen(['python3', 'writer.py'], stdout=subprocess.PIPE)

while 1:
  array = pickle.load(p.stdout)
  array = array.reshape([1, 3, 5, 5, 1])
  print(array.shape)

writer.py:

import numpy as np
import pickle
import sys
while 1:
  array = np.zeros([1, 3, 5, 5, 1], dtype=np.int8)
  pickle.dump(array, sys.stdout.buffer)

answered Mar 6, 2017 at 8:46

ymonad

12.1k1 gold badge41 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I send a byte-object over a pipe in python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related