numpy array - load from disk row by row - memory efficiency but fast

Question

Is there a way to pipeline numpy array from disk, that are saved this way

np.save('data.npy',np.zeros(shape=[500,300,3])) # RGB image

and are read row by row (or column by column) in a similar way like generators work, but without the loading latency?

Detailed description

My application needs near to zero latency, but loading bigger arrays from disk can take some time (~0.02-0.1s). Even this small latency generates unpleasant results.

I have solution for this that satisfies the speed:

dictionary = {'array1': array1, ....}

with this I can immediately access the arrays, but since I am using raspberry pi Zero, my python program is limited with CPU and RAM, so if I have a lot of arrays, I would be dealing with

MemoryError

My application reads the array row by row with frequency 50hz, like this

for row in array:
    [operation with row]
    time.sleep(0.02) # in reality, whole cycle is 0.02s ( including operation time)

I am looking for kind of generator:

def generate_rows(path):
    array = np.load(path)
    for row in array:
        yield row

This solves the problem with memory, but I guess I will lose the near zero latency (loading the array).

Therefore my question is: Is there a way to generate rows like with generator, but the first rows are ready so to say 'immediately', with near zero latency?

EDIT: Based on @Lukas Koestler and @hpaulj comments I tried memmap, but the result is suprisingly not good, because memmap crashes on Memory sooner than simply loading full arrays.

WINDOWS 10

I saved 1000 numpy arrays (shape = [500,30,3]) on the disk and tried to cached them with np.load and np.load with memmap read

import numpy as np
import os

mats = os.listdir('matrixes')
cache = []
for i in range(10):
    for n in mats:
        cache.append(np.load('matrixes\\{}'.format(n),mmap_mode='r')) # Load with memmap
        #cache.append(np.load('matrixes\\{}'.format(n))) #load without memmap

    print('{} objects stored in cache '.format((i+1)*1000))

After running both variants(with memmap and without it), these two errors occured

Memmap after storing 4000 memmaps objects:

...
  File "C:\Python27\lib\site-packages\numpy\core\memmap.py", line 264, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
WindowsError: [Error 8] Not enough memory resources are available to process this command

Simple np.load without memmap after caching 5000 np.arrays

....
File "C:\Python27\lib\site-packages\numpy\lib\format.py", line 661, in read_array
    array = numpy.fromfile(fp, dtype=dtype, count=count)
MemoryError

Raspberry pi Zero

As was pointed out by @Alex Yu, I was testing on windows 10, switching to raspberry pi Zero,

I got above 1000 numpy arrays (took quite long) and then I got

1000 objects stored in cache
Killed

With Memmaps, i got quite quickly above 1000 memmaps, but I got different errors

File "/usr/lib/python2.7/dist-packages/numpy/lib/npyio.py", line 416, in load
    return format.open_memmap(file, mode=mmap_mode)
  File "/usr/lib/python2.7/dist-packages/numpy/lib/format.py", line 792, in open_memmap
    mode=mode, offset=offset)
  File "/usr/lib/python2.7/dist-packages/numpy/core/memmap.py", line 264, in __new__
    mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
mmap.error: [Errno 24] Too many open files

If I am not wrong, this error happens when opening a lot of files, but not closing them.

If you have the liberty of choosing the file format I recommend hdf5 — Mihai Andrei
– Mihai Andrei, Commented Feb 16, 2019 at 23:58
I have a bit of trouble understanding the question. It seems like you want to read a array-file row-by-row in a fast and memory efficient way. However it also seems like you want to operate on 1000+ of these files (simultaneously?). It could be clearer if you elaborated on how the thousands of files fit into the context of the question of reading one file row-by-row. — MSeifert
– MSeifert, Commented Feb 17, 2019 at 11:18

Martin · Accepted Answer · 2019-02-17 19:07:13Z

Thanks to @Lukas Koestler and @hpaulj for directing me into using memmap

and Thanks to @Alex Yu for making the solution reality

Solution to my own question

Using

np.load(path,mmap_mode='r')

works, but is limited by the limit of opened files. On windows and Linux throws different error:

WIN

WindowsError: [Error 8] Not enough memory resources are available to process this command

LIN

mmap.error: [Errno 24] Too many open files

This was solved with the link given by @Alex Yu extend limit of opened files.

Extract:

open

/etc/security/limits.conf

Paste following towards end:

*         hard    nofile      500000
*         soft    nofile      500000
root      hard    nofile      500000
root      soft    nofile      500000

End of Extract

There is still limitation, but it increased the amount up to 8000 objects in list

...
8000 objects stored in cache

until

Traceback (most recent call last):
...
mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
mmap.error: [Errno 12] Cannot allocate memory

For me this is quite enough

General Overview of different attitudes toward my problem

tested on arrays with shape [500,30,3]

1) Simple Load: Without caching

array = np.load(path)
[process rows]

Slowest but most memory efficient

cache_limit = 0 (Arrays in dictionary)

2) Hard cache - loading arrays into dictionary

cache_raw = {i: np.load(i) for i in os.listdir('array_folder')}
...
temporary_array = cache_raw[some_array]
[process rows with temporary_array]

Ultra fast but very memory inefficient

cache_limit ~ 1000, (RPI zero) (Arrays in dictionary)

3) Memmap Cache

cache_memmap = {i: np.load(i,mmap_mode='r') for i in os.listdir('array_folder')}
...
memmap_array = cache_memmap[some_array]
[process rows with memmap_array]

reasonable speed, memory effiecient

cache_limit ~ 8000 (RPI zero) (Arrays in dictionary)

Results

Timing results of loading first row for 20 random accesses for all attitudes:

Memmap
0.00122714042664
0.00237703323364
0.00152182579041
0.000735998153687
0.000724077224731
0.000736951828003
0.000741004943848
0.000698089599609
0.000723123550415
0.000734090805054
0.000775814056396
0.00343084335327
0.000797033309937
0.000717878341675
0.000727891921997
0.000733852386475
0.000690937042236
0.00178194046021
0.000714063644409
0.000691175460815
Hard cache
0.000302076339722
0.000305891036987
0.000910043716431
0.000320911407471
0.000298976898193
0.000309944152832
0.000294923782349
0.000304937362671
0.000298023223877
0.00031590461731
0.000324010848999
0.000273942947388
0.000274181365967
0.000286817550659
0.000277042388916
0.000297784805298
0.000288009643555
0.000318050384521
0.00031304359436
0.000298023223877
Without cache
0.0350978374481
0.0103611946106
0.0172200202942
0.0349309444427
0.0177171230316
0.00722813606262
0.0286860466003
0.0435371398926
0.0261130332947
0.0302798748016
0.0361919403076
0.0286440849304
0.0175659656525
0.035896062851
0.0307757854462
0.0364079475403
0.0258250236511
0.00768494606018
0.025671005249
0.0261180400848

EDIT: Additional computation:

Average time of 100 unique accesses. 5x times for each attitude

Memmap
0.000535547733307 # very good speed
0.000488042831421
0.000483453273773
0.000485241413116
0.00049720287323
Hard cache
0.000133073329926 # 4x faster than memmap
0.000132908821106
0.000131068229675
0.000130603313446
0.000126478672028
Without cache
0.0996991252899 # very slow
0.0946901941299
0.0264434242249 # Interesting to note here, something I suspected
0.0239776492119 # np.load has cache in itself
0.0208633708954 # If you load particular numpy array more times in the program,
#it will load faster. Kind of integrated cache
# From my own experience, it is very unreliable and cannot be counted with.

Great work! I think it could be beneficial for other members of community!

Collectives™ on Stack Overflow

numpy array - load from disk row by row - memory efficiency but fast

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related