UnicodeDecodeError in Python3 when reading binary files stdin

Question

I'm trying to take what's in the stdin and read it 1022 bytes at a time. This code runs fine for text files. But when inputting a binary file it gives me the UnicodeDecodeError. Where data below is sys.stdin.

def sendStdIn(conn, cipher, data):
    while True:
        chunk = data.read(1022)
        if len(chunk)==1022:
            EOFAndChunk = b'F' + chunk.encode("utf-8")
            conn.send(encryptAndPad(cipher,EOFAndChunk))
        else:
            EOFAndChunk = b'T' + chunk.encode("utf-8")
            conn.send(encryptAndPad(cipher,EOFAndChunk))
            break
    return True

The binary file was made by calling dd if=/dev/urandom bs=1K iflag=fullblock count=1K > 1MB.bin

I run the file with essentially python A3C.py < 1MB.bin Then I end up with below.

Traceback (most recent call last):
  File "A3C.py", line 163, in <module>
    main()
  File "A3C.py", line 121, in main
    EasyCrypto.sendStdIn(soc, cipher, sys.stdin)
  File "EasyCrypto.py", line 63, in sendStdIn
    chunk = data.read(1022)
  File "/usr/lib64/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe8 in position 0: invalid continuation byte

Any idea how I can make this so it can read sections of the binary file because I need to send them from this client side to the server side piece at a time. Thanks!

The fix is to code in main(), not in sendStdIn. Good thing you included the full traceback! :-) — Martijn Pieters
– Martijn Pieters, Commented Mar 12, 2017 at 16:53

Martijn Pieters · Accepted Answer · 2017-03-12 16:52:54Z

3

sys.stdin is a text wrapper that decodes binary data. Use sys.stdin.buffer instead:

EasyCrypto.sendStdIn(soc, cipher, sys.stdin.buffer)

The TextIOBase.buffer attribute points to the binary buffered I/O object underneath.

answered Mar 12, 2017 at 16:52

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

N. Hagstrom Over a year ago

Thank you so much! This worked perfectly. It's been driving me mad.

Collectives™ on Stack Overflow

UnicodeDecodeError in Python3 when reading binary files stdin

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related