How to assign a bytes object to a NumPy array slice?

Question

I want to use NumPy arrays to efficiently operate on large byte arrays. Unfortunately, assigning a bytes object to a slice of the NumPy array does not work as I would expect:

import struct
import numpy as np

array = np.zeros(10, dtype=np.uint8)

# Store 65535 into the first two bytes
array[0:2] = struct.pack('<H', 65535)            # does not work

print(array)

This leads to the following exception:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

In this example, the bytes object results from a call to struct.pack. When wrapping the result into a bytearray, everything works as intended, but I think this will perform an unnecessary copy operation (which is not what I want):

array[0:2] = bytearray(struct.pack('<H', 65535)) # works

Why does a bytes object not work here? The fact that bytearray is mutable whereas bytes is not should not make a difference here.

I'm using NumPy version 1.16.4.

cs95 · Accepted Answer · 2019-06-12 17:32:38Z

3

You can create a memoryview which references the underlying buffer. This should avoid creating a copy.

array[:2] = memoryview(struct.pack('<H', 65535))
array
# array([255, 255,   0,   0,   0,   0,   0,   0,   0,   0], dtype=uint8)

It seems you cannot assign a byte string directly as NumPy attempts to decode the bytes into characters according to an encoding scheme (which would explain the UnicodeDecodeError) before assignment. With a bytearray or memoryview, it is implied that each byte is separate.

edited Jun 12, 2019 at 17:32

answered Jun 12, 2019 at 17:05

cs95

406k106 gold badges744 silver badges797 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ph4nt0m Over a year ago

According to the docs both bytes and bytearray do support the buffer protocol. So there must be some other difference, maybe it is the numpy implementation itself?

cs95 Over a year ago

@ph4nt0m Only speculating here, I think the reason is that if you pass a byte string v/s a bytearray, numpy will try to decode the former according to a character encoding scheme. Whereas in a bytearray, the bytes are interpreted separately. Just my guess.

Collectives™ on Stack Overflow

How to assign a bytes object to a NumPy array slice?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related