Python multiprocessing: calling pool.map within a function

Question

I am trying to use the mutltiprocessing package to use multiple CPUs within a function. When I run a toy example outside of a function it runs in a quarter of a second with no problems (see below).

from multiprocessing import Pool
import time

start = time.clock()

def f(x):
    return x*x

if __name__ == '__main__':
    with Pool(processes=7) as pool:     
        result = pool.map(f, range(1000))



print(time.clock() - start)

However, when I adapt the same code into a function (see below), it prints True to indicate that __name__ == '__main__', but then it runs forever and never returns a result. I am running Python 3.3 on Windows 7.

from multiprocessing import Pool
import time

start = time.clock()

def f(x):
    return x*x

def testfunc(r):
    if __name__ == '__main__':
        print(True)
        with Pool(processes=7) as pool:     
            result = pool.map(f, range(r))

    return result

result = testfunc(1000)
print(time.clock() - start)

Try putting the if __name__ == '__main__' block outside the function, around the call to testfunc. — BrenBarn
– BrenBarn, Commented Sep 25, 2013 at 17:36
In all of the three code snippets mentioned (2 above, 1 in answer), the usage of "with" gives an error. It is the AttributeError: __exit__. I cannot even get the first example to run... Am I missing something? — Pavithran Iyer
– Pavithran Iyer, Commented Apr 2, 2015 at 23:53

falsetru · Accepted Answer · 2013-09-25 17:36:57Z

5

You are using if __name__ == '__main__' in wrong place.

from multiprocessing import Pool
import time

start = time.clock()

def f(x):
    return x*x

def testfunc(r):
    print(True)
    with Pool(processes=7) as pool:     
        result = pool.map(f, range(r))
    return result

if __name__ == '__main__':
    result = testfunc(1000)
    print(time.clock() - start)

According to multiprocessing - Programming guidelines:

Safe importing of main module

Make sure that the main module can be safely imported by a new Python interpreter without causing unintended side effects (such a starting a new process).

... one should protect the “entry point” of the program by using if __name__ == '__main__': as follows:

answered Sep 25, 2013 at 17:36

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Pavithran Iyer Over a year ago

Have you tested this code? It gives an error saying AttributeError: __exit__

falsetru Over a year ago

@PavithranIyer, Which version of python are you using? I tested this with Python 3.3 at the time of posting. I think you're using python 2.x. in which multiprocessing.Pool is not a context manager. asciinema.org/a/18325

falsetru Over a year ago

@PavithranIyer, If you're using Python 2.x, try following: pastebin.com/hHKbgaSc (without using context manager)

Collectives™ on Stack Overflow

Python multiprocessing: calling pool.map within a function

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related