Memory error while converting list to numpy array

Question

I've got a total of around 7000 images from which I'm extracted HoG features. I then want to convert the list into an np array for further processing. But I get a memory error during the convertion.

Here's the relevant part of my code:

from skimage import data, io, filter, color, exposure
from skimage.feature import hog
from skimage.transform import resize
import matplotlib.pyplot as plt
import numpy as np

tmp_hogs = [] # this is the list I need to convert into a numpy array
for group in samplegroups:
    for myimg in group:
        curr_img = np.array(myimg, dtype=float)
        imgs.append(curr_img)
        fd, hog_image = hog(curr_img, orientations=8, pixels_per_cell=(4, 4),
                 cells_per_block=(1, 1), visualise=True, normalise=True)
        tmp_hogs.append(fd)

img_hogs = np.array(tmp_hogs, dtype =float)

The error I get is:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Users\app\anacondasoftware\lib\threading.py", line 810, in __bootstrap_inner
    self.run()
  File "C:\Users\app\anacondasoftware\lib\site-packages\spyderlib\widgets\externalshell\monitor.py", line 582, in run
    already_pickled=True)
  File "C:\Users\app\anacondasoftware\lib\site-packages\spyderlib\utils\bsdsocket.py", line 45, in write_packet
    nsend -= temp_fail_retry(socket.error, sock.send, sent_data)
  File "C:\Users\app\anacondasoftware\lib\site-packages\spyderlib\utils\bsdsocket.py", line 25, in temp_fail_retry
    return fun(*args)
error: [Errno 10054] An existing connection was forcibly closed by the remote host

Traceback (most recent call last):
  File "C:\Users\app\Documents\Python Scripts\gbc_carclassify.py", line 63, in <module>
    img_hogs = np.array(tmp_hogs, dtype =float) 
MemoryError

How can I fix it?

How much memory do you have? How much memory does one hog occupy? Are you using 32 or 64-bit python? — DrV
– DrV, Commented Jun 21, 2014 at 17:34

Saullo G. P. Castro · Accepted Answer · 2014-06-21 14:02:35Z

5

For RGB or RGBA images you need only 8 bits per value, and when using float you are alocating 64 bits per value. Try using np.uint8 instead:

img_hogs = np.array(tmp_hogs, dtype=np.uint8)

edited Jun 21, 2014 at 14:02

answered Jun 21, 2014 at 13:53

Saullo G. P. Castro

59.4k28 gold badges191 silver badges244 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Saullo G. P. Castro Over a year ago

another improvement will be to create an empty array and store the images in this array directly, avoiding the intermediate list

Collectives™ on Stack Overflow

Memory error while converting list to numpy array

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related