multithreading/processing in Python

Question

Hello I am trying define values of a dictionary in parallel using multiprocessing. When the function f() is called outside "pool" the dictionary value is set correctly. In the pool call however it fails.

What am I doing wrong? Thanks.

from multiprocessing import Pool

hits={}
def f(x):
    hits[x] = x  #this will involve a complex operation

f('000')
print hits['000']

if __name__ == '__main__':
    pool = Pool(processes=2)             
    inputs = ['a','b','c']
    result = pool.map(f, inputs)
print hits['a']

Sharing state between processes is hard. Perhaps perform the complex operation in child processes and maintain the global state of hits in the parent. — Tim McNamara
– Tim McNamara, Commented Oct 25, 2011 at 21:57

Sven Marnach · Accepted Answer · 2011-10-25 21:56:40Z

7

You spawn a bunch of subprocesses which inherit the current value of hits. Each subprocess get its own copy of hits and modifies it, then exits, dropping the process-local copy of hits.

The intended use of multiprocessing.Pool.map() is to use its return value, which you are ignoring. Each process could return its copy of hits, and you finally join these copies.

answered Oct 25, 2011 at 21:56

Sven Marnach

608k123 gold badges966 silver badges865 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

David Over a year ago

I see, then the answer is: from multiprocessing import Pool hits={} def f(x): return x if name == 'main': pool = Pool(processes=2) inputs = ['a','b','c'] result = pool.map(f, inputs) result=dict(zip(result,result)) print result.items()

Collectives™ on Stack Overflow

multithreading/processing in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related