Using Global Variables in Multiprocessing Pool

Question

I am having issues with trying to use global variables in a multiprocessing pool. I know there is quite a lot of content on this already, but I am struggling to implement the answers to my case.

I have a function that I want to use with pool.map. I want the input of this function to be a few global variables and also one other parameter that changes process-to-process. That is I want something that looks like:


if __name__ == '__main__':

   A = Global Variable Array
   B = Global Variable Array
   C = Actual Input Array For the Multiprocessing

   pool = multiprocessing.Pool()
   Res= pool.map(Function,Input = A,B,C)

In the above case only C varies from process-to-process. Currently my code only works if I import the global arrays from an external files within each process, which seems very wasteful.

"I want something that looks like" What stopy you from writing something that looks like this? What have you tried so far? Did you get an error? What do you intend Input = A, B, C to mean? Pool.map does not take a parameter Input, and Input = A, B, C passes only A to Input. — MisterMiyagi
– MisterMiyagi, Commented Jul 12, 2020 at 13:27

alex_noname · Accepted Answer · 2020-07-12 13:25:55Z

1

multiprocessing provides a couple of ways to share state between processes. Access to these objects may be synchronized by lock arg.

If lock is True (the default) then a new recursive lock object is created to synchronize access to the value. If lock is a Lock or RLock object then that will be used to synchronize access to the value. If lock is False then access to the returned object will not be automatically protected by a lock, so it will not necessarily be “process-safe”.

Example from the official documentation:

from multiprocessing import Process, Value, Array

def f(n, a):
    n.value = 3.1415927
    for i in range(len(a)):
        a[i] = -a[i]

if __name__ == '__main__':
    num = Value('d', 0.0)
    arr = Array('i', range(10))

    p = Process(target=f, args=(num, arr))
    p.start()
    p.join()

    print(num.value)
    print(arr[:])

edited Jul 12, 2020 at 13:25

answered Jul 12, 2020 at 13:20

alex_noname

33.2k6 gold badges95 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using Global Variables in Multiprocessing Pool

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related