I am trying to perform mean scaling on a bunch of images (>40k). When I read in the images, of size (3,256,256), into an np array, memory usage is at %40 (out of 60 GB, checked using htop). However, when I run arr.std(), the program crashes and gives a MemoryError, even though usage is still at %40.
Any thoughts on what might be wrong?
Are you completly sure that each cell of your arrays takes only 1 Byte, because maybe by default it allocates 8 Bytes for cell.
I created small array 3 x 3 and it occupies 72 bytes.
import numpy as np
a = np.array(np.mat('1, 2, 3; 4, 5, 6; 7, 8, 9'))
print(a.nbytes) # Use this .nbytes instead of sys.getsizeof
256 x 256 x 3 x 8 Bytes = 1572864 B = 1.5 MB
1.5 MB x 40,000 = 60000 MB \approx 58.6 GB
And you said that you have at least 40 thousands, so if you have more that that and std is using some memory to flatten the array (see http://docs.scipy.org/doc/numpy-1.9.2/reference/generated/numpy.std.html and you would land here https://github.com/numpy/numpy/blob/master/numpy/core/_methods.py) you would run out of memory.
The solution is very simple: Enforce the byte type int8 or another from here: http://docs.scipy.org/doc/numpy-1.9.2/user/basics.types.html
a = np.array(np.mat('1, 2, 3, ; 4, 5, 6; 7, 8, 9'), dtype=np.int8)
print(a.nbytes) # Only 9 Bytes
to check available memory try pythonic way (instead of htop):
import psutil
m = psutil.virtual_memory()
print(m.available)
P.S. Remember that array.nbytes shows the amount of memory consumed only by array elements without some auxiliary bytes for array maintenance.
arr.std()try to allocate a single big chunk of memory? If so, and the requested size is larger than the remaining memory, the request would fail and the memory usage would remain unchanged...