python memory intensive script

Question

After about 4 weeks of learning, experimenting, etc. I finally have a script which does what I want. It changes the perspective of images according to a certain projection matrix I have created. When I run the script for one image it works fine, however I would like to plot six images in one figure. When I try to do this I get a memory error.

All the images are 2448px in width and 2048 px in height each. My script:

files = {'cam1': 'c1.jpg',
         'cam2': 'c2.jpg',
         'cam3': 'c3.jpg',
         'cam4': 'c4.jpg',
         'cam5': 'c5.jpg',
         'cam6': 'c6.jpg'}

fig, ax = plt.subplots()

for camname in files:
    img = Image.open(files[camname])
    gray_img = np.asarray(img.convert("L"))
    img = np.asarray(img)
    height, width, channels = img.shape

    usedP = np.array(P[camname][:,[0,1,3]])
    usedPinv = np.linalg.inv(usedP)
    U, V = np.meshgrid(range(gray_img.shape[1]),
                       range(gray_img.shape[0]))
    UV = np.vstack((U.flatten(),
                    V.flatten())).T
    ones = np.ones((UV.shape[0],1))
    UV = np.hstack((UV, ones))

    # create UV_warped
    UV_warped = usedPinv.dot(UV.T).T

    # normalize vector by dividing by the third column (which should be 1)
    normalize_vector = UV_warped[:,2].T
    UV_warped = UV_warped/normalize_vector[:,None]

    # masks
    # pixels that are above the horizon and where the V-projection is therefor positive (X in argus): make 0, 0, 1
    # pixels that are to far: make 0,0,1
    masks = [UV_warped[:,0]<=0, UV_warped[:,0]>2000, UV_warped[:,1]>5000, UV_warped[:,1]<-5000] # above horizon: => [0,0,1]
    total_mask = masks[0] | masks[1] | masks[2] | masks[3]
    UV_warped[total_mask] = np.array([[0.0, 0.0, 1.0]])

    # show plot
    X_warped = UV_warped[:,0].reshape((height, width))
    Y_warped = UV_warped[:,1].reshape((height, width))
    gray_img = gray_img[:-1, :-1]

    # add colors
    rgb = img[:,:-1,:].reshape((-1,3)) / 255.0 # we have 1 less faces than grid cells
    rgba = np.concatenate((rgb, np.ones((rgb.shape[0],1))), axis=1)

    plotimg = ax.pcolormesh(X_warped, Y_warped, img.mean(-1)[:,:], cmap='Greys')
    plotimg.set_array(None)
    plotimg.set_edgecolor('none')
    plotimg.set_facecolor(rgba)

ax.set_aspect('equal')
plt.show()

I have the feeling that numpy.meshgrid is quite memory intensive, but I'm not sure. Does anybody see where my memory gets eaten away rapidly? (BTW, I have a laptop with 12Gb of RAM, which is only used by other programs for a very small part)

Python as a whole, especially in memory-intensive applications, is much slower than compiled languages like C/C++. Interpreted languages like Python aren't as efficient in memory management. — signus
– signus, Commented Mar 13, 2014 at 13:15
You are right. However, changing to C/C++ is not an option at this point in time. — Yorian
– Yorian, Commented Mar 13, 2014 at 13:30

Germano · Accepted Answer · 2014-03-13 13:00:24Z

2

You might want to profile your code with this library.

It will show you where your script is using memory.

answered Mar 13, 2014 at 13:00

Germano

2,48218 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Yorian Over a year ago

Thanks for your reply, I'm however not really a hero when it comes to using the command prompt... I tried it but I haven't got it working yet

Yorian Over a year ago

Thanks for your response, I think I found the problem. The script loads the images using float64 types which seem to use a lot of memory.

Community · Accepted Answer · 2017-05-23 12:17:05Z

2

There is a Stackoverflow question here about memory profilers. Also, I've used the trick in this answer in the past as a quick way to get an idea where in the code memory is going out of control. I just print the resource.getrusage() results all over the place. It's not clean, and it doesn't always work, but it's part of the standard library and it's easy to do.

edited May 23, 2017 at 12:17

CommunityBot

11 silver badge

answered Mar 13, 2014 at 15:19

kristofor

866 bronze badges

Comments

signus · Accepted Answer · 2014-03-13 13:34:56Z

1

I ordinarily profile with the profile and cProfile modules, as it makes testing individual sections of code fairly easy.

Python Profilers

answered Mar 13, 2014 at 13:34

signus

1,14815 silver badges47 bronze badges

Collectives™ on Stack Overflow

python memory intensive script

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related