Reading a numpy array of strings in memmap mode

Question

I created this numpy array and stored it in disk as follows:

s = (b'foo', b'bar', b'baz', b'buzz')
def build_numpy_array():
  return np.fromiter((s for _ in range(200)), dtype=[('foo','S40'), 
  ('bar', 'S40'), ('baz', 'S40'), 
  ('buzz', 'S40')
  ])

np.save('data.dat', {'data': build_numpy_array()})

This works fine np.load('data.dat.npy')

But, I want to use it in memmap mode. So this fails

np.load('data.dat.npy',mmap_mode='r')

ValueError: Array can't be memory-mapped: Python objects in dtype.

And this gives weird encoding

np.memmap('data.dat.npy',  mode='r',dtype=[('foo','S40'), 
  ('bar', 'S40'), ('baz', 'S40'), 
  ('buzz', 'S40')
  ])

 (b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00bar', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00baz', b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00buz', b'z\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00foo')

hpaulj · Accepted Answer · 2019-12-10 19:18:48Z

1

data = build_numpy_array()

is a (200,) structured array.

to load:

In [152]: np.save('data.dat', {'data': data})

I have to use allow_pickle:

In [157]: x=np.load('data.dat.npy',allow_pickle=True)

x is () shaped object array. That is x.item() is a dictionary, containing the array as an element value.

The problem lies with the save - it's saving a dictionary.

In [161]: np.save('data.dat', data)                                             
In [162]: x=np.load('data.dat.npy')                                             
In [163]: x.shape                                                               
Out[163]: (200,)

now

In [165]: r = np.load('data.dat.npy',mmap_mode='r')                             
In [166]: r                                                                     
Out[166]: 
memmap([(b'foo', b'bar', b'baz', b'buzz'),
        (b'foo', b'bar', b'baz', b'buzz'),
        (b'foo', b'bar', b'baz', b'buzz'),
    ...

edited Dec 10, 2019 at 19:18

answered Dec 10, 2019 at 19:16

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

frazman Over a year ago

Thanks for the response. I see the issue. But is the load happening in memory map mode?

hpaulj Over a year ago

Apparently memory map mode cannot handle the object dtype, the pickle. But it works if we simply save data, rather than a dictionary containing data. The dictionary works with: np.savez('data.dat', **{'data': data}) (but that's a zip archive).

Collectives™ on Stack Overflow

Reading a numpy array of strings in memmap mode

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related