Using multithreading for maximum CPU efficiency

Question

I am currently working in Python and my program looks like this:

function(1)
function(2)
...
function(100)

Performing a function takes ~30 minutes at 100% CPU, so executing the program takes a lot of time. The functions access the same file for inputs, do a lot of math and print the results.

Would introducing multithreading decrease the time, which the program takes to complete (I am working on a multicore machine)? If so, how many threads should I use?

Thank you!

It depends on where the bottleneck is. Are you repeatedly reading/writing from files? Can you do all the reading/writing at once? — Joel Cornett
– Joel Cornett, Commented Jul 26, 2012 at 16:15
It depends on what those functions are doing also. If function2 is dependent on the output from function1, there isn't much you can do. There are some good schedulers out there (not sure what there is for python in particular) which do a good job of allowing multiple jobs to run in parallel, but it really depends on where the dependencies and bottlenecks are. — Liron
– Liron, Commented Jul 26, 2012 at 16:17

atomicinf · Accepted Answer · 2012-07-26 16:18:15Z

1

It depends.

If none of the functions depend on each other at all, you can of course run them on separate threads (or even processes using multiprocessing, to avoid the global interpreter lock). You can either run one process per core, or run 100 processes, or any number in between, depending on the resource constraints of your system. (If you don't own the system, some admins don't like users who spam the process table.)

If the functions must be run one after the other, then you can't do that. You have to restructure the program to try and isolate independent tasks, or accept that you might have a P-complete (inherently hard to parallelize) problem and move on.

answered Jul 26, 2012 at 16:18

atomicinf

3,76621 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user1492295 Over a year ago

The functions are completely independent.

atomicinf Over a year ago

Then you'll want to look at docs.python.org/library/multiprocessing.html - that module should have what you're looking for. As long as your task is CPU-bound and not IO-bound, you should see some speedup.

user1277476 Over a year ago

+1 multiprocessing. As far as the number of processes to use, assuming you're CPU-bound, use as many as you have cores.

Collectives™ on Stack Overflow

Using multithreading for maximum CPU efficiency

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related