convert binary string to numpy array

Question

Assume I have the string:

my_data = '\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@'

Where I got it is irrelevant, but for the sake of having something concrete, assume I read it from a binary file.

I know my string is the binary representation of 4 (4-byte) floats. I would like to get those floats as a numpy array. I could do:

import struct
import numpy as np
tple = struct.unpack( '4f', my_data )
my_array = np.array( tple, dtype=np.float32 )

But it seems silly to create an intermediate tuple. Is there a way to do this operation without creating an intermediate tuple?

EDIT

I would also like to be able to construct the array in such a way that I can specify the endianness of the string.

possible duplicate of How do I create a numpy array from string? — Aurelius
– Aurelius, Commented Feb 6, 2015 at 18:02
@Aurelius I would say this is close, but not an exact duplicate. Though the answers are similar, this question is about floats and that question is about integers. — ohmu
– ohmu, Commented Feb 6, 2015 at 19:29

mgilson · Accepted Answer · 2020-12-08 06:38:55Z

51

>>> np.frombuffer(b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@', dtype='<f4') # or dtype=np.dtype('<f4'), or np.float32 on a little-endian system (which most computers are these days)
array([ 1.,  2.,  3.,  4.], dtype=float32)

Or, if you want big-endian:

>>> np.frombuffer(b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@', dtype='>f4') # or dtype=np.dtype('>f4'), or np.float32  on a big-endian system
array([  4.60060299e-41,   8.96831017e-44,   2.30485571e-41,
         4.60074312e-41], dtype=float32)

The b isn't necessary prior to Python 3, of course.

In fact, if you actually are using a binary file to load the data from, you could even skip the using-a-string step and load the data directly from the file with numpy.fromfile().

Also, dtype reference, just in case: http://docs.scipy.org/doc/numpy/reference/arrays.dtypes.html

edited Dec 8, 2020 at 6:38

mgilson

312k70 gold badges656 silver badges722 bronze badges

answered Aug 1, 2012 at 13:19

JAB

21.2k6 gold badges73 silver badges80 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

mgilson Over a year ago

This is excellent. (thanks). One downside here that I can see is that there's no way to specify endianness. Any ideas on that one?

JAB Over a year ago

Specify the endianness in the dtype. np.dtype('<f4') for little-endian (though this is the default, so isn't necessary for anything other than code clarity), np.dtype('>f4') for big-endian. So np.fromstring(b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@', dtype=np.dtype('>f4')) results in array([4.60060299e-41, 8.96831017e-44, 2.30485571e-41, 4.60074312e-41], dtype=float32). Reference: docs.scipy.org/doc/numpy/reference/arrays.dtypes.html

mgilson Over a year ago

perfect. just what I was looking for.

mgilson Over a year ago

I think you should add the stuff about endianness conversion to your answer. I think that could also be potentially helpful to someone else with the same question I had so it makes sense to feature it more prominently than in a comment.

mtrw Over a year ago

@JAB - +1. One note - I don't think little-endian is the default. Rather, machine type is the default. So if you're on a big endian machine (rare these days) and are reading a little endian string, you need to specify '<f4'.

|

yoonghm · Accepted Answer · 2020-12-05 23:38:04Z

0

np.fromstring() is deprecated. Use np.frombuffer() instead.

import numpy as np

my_data = b'\x00\x00\x80?\x00\x00\x00@\x00\x00@@\x00\x00\x80@'

# np.fromstring is deprecated
# data = np.fromstring(my_data, np.float32)
data = np.frombuffer(my_data, np.float32)

print(data)

[1. 2. 3. 4.]

answered Dec 5, 2020 at 23:38

yoonghm

4,6851 gold badge40 silver badges53 bronze badges

Collectives™ on Stack Overflow

convert binary string to numpy array

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related