3

I am a newbie programmer, so a little lost here.

I've written a small python function that is rather time consuming to run depending on my search area, which is expected.

After a certain threshold, about a million records my program always seems to get lost. Python shell is not responding, but CPU usage remains contstant at about 13%. Also I can't stop the program execution by KeyboardInterrupt

So my question: Is it possible to increase the CPU usage for my python program to make it run faster?

A side question: Any help on why python shell becomes unresponsive? I am not using excessive memory, I just need to be able to cycle through a large set as quickly as possible.

Improve this question
2
  • 3
    Python shell does not become unresponsive just like that. It is your code that makes it unresponsive. The same goes for CPU utilization - your code does not utilize it. So the problem is not in Python itself: it is in your code. Show us the code and we'll try to solve the problem. Commented Jan 19, 2014 at 20:13
  • The bottleneck is more likely to be memory. Are you doing something silly (like: loading those "million" records at the same time, provided that they contain big data)? Without you showing the code, we can only speculate about the problem. Commented Jan 19, 2014 at 20:40

1 Answer 1

Reset to default
3

Is your cpu a multicore cpu? If yes there are several ways to use multiple cores with python.

Built-in is the multiprocessing module. The multiprocessing.Pool class provides vectorization across multiple CPUs with the map() and related methods. There is a trade-off in here though. If you have to communicate large amounts of data between the processes, that overhead might negate the advantage of multiple cores. Use a suitable build of numpy. If numpy is built with a multithreading ATLAS library, it will be faster on large problems. Use extension modules like numexpr, parallel python, corepy or Copenhagen Vector Byte Code.

Note that the threading module isn't all that useful in this regard. To keep memory management simple, the global interpreter lock ("GIL") enforces that only one thread at a time can be executing python bytecode. External modules like numpy can use multiple threads internally, though.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

In particular, if the program pegs 1 of 8 cores, that would be ~12.5% CPU usage.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.