Run a python program on multiple cores

Question

I am new to multi-core programming. The following program using only one core. How can I make it to run on multiple cores (I have 4 cores).

simDict={}

def sim(outer,inner, ...):
    val= /*do some math*/
    simDict[...]=val

def foo():
   for outer in xrange(0, limit):
      for inner in xrange(outer, limit):
          sim(outer,inner, ...)
foo()

nneonneo · Accepted Answer · 2012-10-21 05:29:39Z

5

Easy:

from multiprocessing import Pool

p = Pool()

def do_inner(outer, limit):
    for inner in xrange(outer, limit):
        sim(outer, inner, ...)

def foo():
    p.map(do_inner, xrange(limit))

foo()

This employs multiprocessing.Pool to create a pool of worker processes.

answered Oct 21, 2012 at 5:29

nneonneo

181k37 gold badges331 silver badges412 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

sean · Accepted Answer · 2012-10-21 05:30:16Z

1

For your problem I would use a queue then I would stick with the consumer/producer problem and go from there. Unless there is some data dependency between your threads then you will need to get into some more advanced locking mechanisms and have to protect against far more threading pitfalls.

Producer/Consumer concept: http://en.wikipedia.org/wiki/Producer-consumer_problem And just for reference: http://docs.python.org/library/multiprocessing.html

multiprocessing may be helpful to use as well as the global interpreter lock can slow things down in some specific cases if just threads are used.

answered Oct 21, 2012 at 5:30

sean

3,98523 silver badges28 bronze badges

2 Comments

nneonneo Over a year ago

He explicitly wants to run on multiple cores, and the algorithm looks CPU-bound. Therefore, multiprocessing is the only choice if he wants to stay with pure-Python programming.

sean Over a year ago

Agreed, why I put the link to the multiprocessing at the end.

Collectives™ on Stack Overflow

Run a python program on multiple cores

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related