Releasing memory of huge numpy array in IPython

Question

UPDATE:- This problem solved itself after a machine reboot. Not yet able to figure out why this error was happening before.

I have a function that loads a huge numpy array (~ 980MB) and returns it.

When I first start Ipython and call this function, it loads the array into the variable without any problem.

But if I run the same command again, it exits raising a "Memory Error".

I tried the following,

del hugeArray

Still the same error was occurring. I even tried the following

del hugeArray
gc.collect()
gc.collect()

Initially, gc.collect() returned 145 and the second call returned 48. But even after this when I call the function, it was still raising a Memory error.

The only way I could load again was to restart ipython. Is there something I can do to free all memory in ipython, so that I don't have to restart it?

----------------Update

Following is the output of %whos

Variable   Type      Data/Info
------------------------------
gc         module    <module 'gc' (built-in)>
gr         module    <module 'Generate4mRamp' <...>rom 'Generate4mRamp.pyc'>
np         module    <module 'numpy' from '/us<...>ages/numpy/__init__.pyc'>
plt        module    <module 'matplotlib.pyplo<...>s/matplotlib/pyplot.pyc'>

Out of this, gr is my module containing the function which i used to load the data cube.

---------How to Reproduce the error

The following simple function is able to reproduce the error.

import numpy as np
import gc

def functionH():
    cube=np.zeros((200,1024,1024))
    return cube

testcube=functionH()   #Runs without any issue 

del testcube
testcube=functionH()  # Raises Memory Error

del testcube
gc.collect()
gc.collect()
testcube=functionH()  # Still Raises Memory Error

This error is occurring only in Ipython. In simple python (>>>) after giving del testcube, there is no Memory Error.

Can you try calling whos in ipython to find out what is taking up memory? — tiago
– tiago, Commented Apr 28, 2013 at 9:15
@tiago : I have added the output of the %whos command. It doesn't show any thing other than the modules i loaded. — indiajoe
– indiajoe, Commented Apr 28, 2013 at 9:52
Can you show the reference count of the object before delete it? import sys;sys.getrefcount(testcube) — HYRY
– HYRY, Commented Apr 28, 2013 at 11:34
Hey, All of a sudden it started working now. after del command it is able to run the function again without Memory Error. I shall try to see what change happened in between. — indiajoe
– indiajoe, Commented Apr 28, 2013 at 11:55

Thomas K · Accepted Answer · 2013-04-29 11:55:50Z

63

Are you looking at the value? IPython caches output variables as e.g. Out[8], so if you examine it, it will be kept in memory.

You can do %xdel testcube to delete the variable and remove it from IPython's cache. Alternatively, %reset out or %reset array will clear either all your output history, or only references to numpy arrays.

answered Apr 29, 2013 at 11:55

Thomas K

40.7k7 gold badges88 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

indiajoe Over a year ago

I wasn't looking at the value in IPython. But thanks for this info about the Ipython caching the outputs. I didn't know about this. I shall try out this xdel command also when this error occurs next time. Right now, everything is working fine.

Kobe-Wan Kenobi Over a year ago

Is there a reason for this not to work properly? I try doing something like list([i**2 for i in range(30000000)]) to test this, and then I try to do the %reset out. Out is empty after that, but the memory is still occupied as before the reset command, why could that be? It's like it's only removing the reference. but not releasing memory?

Kobe-Wan Kenobi Over a year ago

I should add that %reset, without out does the work. It both empties the memory and removes the references. But I don't want to loose my own variables. Also, this works v = Out[18] %xdel v. Why is that?

Thomas K Over a year ago

%xdel tries to find and remove all the references to an object in IPython's machinery.

Collectives™ on Stack Overflow

Releasing memory of huge numpy array in IPython

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related