Global variable access during Python multiprocessing

Question

name two lists A, B

I'd like to delete the B list elements from A list.

Python code not using multiprocessing

A = ["leo", "kiki", "eden"]
B = ["eden", "kiki"]

for i in B:
    A.remove(i)

The multi-processing code that I thought of is as follows.

from multiprocessing import Pool
import time
A = ["leo", "kiki", "eden"]
B = ["eden", "kiki"]


def test(i):
    global A
    A.remove(i)
    print("intest : ",A)




if __name__ == '__main__':
    global A
    pool = Pool(processes=2)
    pool.map(test ,B)
    pool.close()
    pool.join()
    print("final : ",A)

output results :

intest :  ['leo', 'kiki']
intest :  ['leo']
final :  ['leo', 'kiki', 'eden']

why in "intest" The global variable applies correctly.

in "final" the global variable change is not applies?

Please give me a lot of help.

They're separate processes with separate memory; regular variables aren't magically copied between the processes. That magic is reserved only for the special multiprocessing types such as Queues and shared_memory blocks. — AKX
– AKX, Commented Jul 14, 2021 at 7:23
With that in mind, we'd really need to know what you'd actually be doing with multiprocessing. I think your question here is maybe oversimplified? :) — AKX
– AKX, Commented Jul 14, 2021 at 7:24
@AKX Thank you for your reply :) I will search some multiprosessing queues and shared_memory!!! — Lumia
– Lumia, Commented Jul 14, 2021 at 7:30

JiyuuSensei · Accepted Answer · 2021-07-14 07:56:38Z

3

Global variables are not shared between processes. When you create a new process, it is a different python instance that is launched. They cannot share values, they are just copied on creation. Any adjustments you do on the child process will only be visible to that process, not your original python process that launched them.

To get around this you can use managers from the multiprocessing module. You can create a manager.list() which allows values to be shared between processes. The following example removed the global parameters and makes use of the Manager class:

import time
from multiprocessing import Pool
from multiprocessing import Manager


def test(test_list,i):
    test_list.remove(i)
    print("intest : ", test_list)
    

if __name__ == '__main__':
    with Manager() as manager:
        A = manager.list(["leo", "kiki", "eden"])
        B = ["eden", "kiki"]

        print("Start : ", A)
        pool = Pool(processes=2)
        pool.starmap(test, [(A, K) for K in B])
        pool.close()
        pool.join()
        print("Final : ", A)

edited Jul 14, 2021 at 7:56

answered Jul 14, 2021 at 7:31

JiyuuSensei

5725 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Lumia Over a year ago

I ran the code you posted. There was an little error.(proceesing global variable A in test(i)

Lumia Over a year ago

So I edit little bit and it's working very well ^^ That was a lot of help! using managers def test(A,i): A.remove(i) print("intest : ", A) if name == 'main': with Manager() as manager: A = manager.list(["leo", "kiki", "eden"]) B = ["eden", "kiki"] print("Start : ", A) pool = Pool(processes=2) pool.starmap(test, [(A, K) for K in B]) pool.close() pool.join() print("Final : ", A)

JiyuuSensei Over a year ago

You're correct, i edited the answer accordingly. I must have wrongly copied my test function, it should contain an extra argument. I've also adjusted pool.map() to pool.starmap(), which is indeed more correct in this scenario. Thanks!

xaostheory · Accepted Answer · 2021-07-14 07:35:13Z

1

sharing memory by using global is enough when multithreading. Sharing memory on multiprocessing has to be implemented in a different way and that's because different processes work with different "copies" of your variables and objects. Sharing memory between processes can be achieved by using the SharedMemory Class of multiprocessing module which also provides the ability to create list-like objects that can be shared between processes, by using the ShareableList class. So you should transform your code that way:

from multiprocessing import shared_memory
A = shared_memory.ShareableList(["leo", "kiki", "eden"])
# you must always close and unlink shared_memory
A.shm.close()
A.shm.unlink()

You can read more about this on the official python documentation: https://docs.python.org/3/library/multiprocessing.shared_memory.html

answered Jul 14, 2021 at 7:35

xaostheory

1035 bronze badges

1 Comment

Lumia Over a year ago

Thanks for your help. The Python version is low. I haven't tried it yet, but I'll try it. ^^

Collectives™ on Stack Overflow

Global variable access during Python multiprocessing

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related