Limit Python process memory usage

Question

I have a system with 16GB of memory. I run a python script for some data mining application and the process takes up the entire 16GB. I want to limit the python process to take up only a limited amount of memory.

Is it possible to do this? If yes, how?

Update:

The application retrieves hotel review data from a huge database and tries to build graphs among hotels and users for some analysis. The data structure to hold the data goes beyond 16GB.

Could you possibly post some example code so that we can be of more use? — Slater Victoroff
– Slater Victoroff, Commented Jul 11, 2012 at 21:56
On *nix, use ulimit (see stackoverflow.com/questions/1760025/limit-python-vm-memory) — hcarver
– hcarver, Commented Jul 11, 2012 at 21:56
There are various ways to restrict the amount of total memory allowed to be used (e.g., ulimit), but this will cause the application to fail when it reaches the limit. It is a design flaw of the application if it is taking up 16GB of memory, and you haven't given us any details of the application, and therefore we don't have enough information to help you solve it. — Jay Sullivan
– Jay Sullivan, Commented Jul 11, 2012 at 21:58
You could check something like this: lshift.net/blog/2008/11/14/tracing-python-memory-leaks — user1277476
– user1277476, Commented Jul 11, 2012 at 23:13
@notfed: I have updated the question with more detail on the application! Thanks. — bdhar
– bdhar, Commented Jul 14, 2012 at 17:25

Jay Sullivan · Accepted Answer · 2012-07-16 20:44:58Z

2

You have an application which has a data structure that appears to require 16GB+ of RAM, right? If so, then limiting its RAM will cause the application to fail, will it not?

Is there a reason all of this data needs to be in RAM all at once? Likely, the application could be redesigned to take up less than 16GB of RAM at a time. For example, searching all file contents of all files on a computer does not require having all files open in RAM all at once, but only one at a time.

My point is that "limiting the RAM usage" isn't likely to be solvable any way other than redesigning the application.

answered Jul 16, 2012 at 20:44

Jay Sullivan

18.4k11 gold badges67 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

gspr Over a year ago

It could be that his algorithm is hard to bound space-wise, and he just wants the process killed before his computer starts hitting swap. In that case, he should look at ulimit, as has been mentioned.

Collectives™ on Stack Overflow

Limit Python process memory usage

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related