1

I have a function in which I create a pool of processes. More over I use multiprocessing.Value() and multiprocessing.Lock() in order to manage some shared values between processes.

I want to do the same thing with an array of objects in order to share it between processes but I don't know how to do it. I will only read from that array.

This is the function:

from multiprocessing import Value,Pool,Lock,cpu_count

def predict(matches_path, unknown_path, files_path, imtodetect_path, num_query_photos, use_top3, uid, workbook, excel_file_path,modelspath,email_address):
 
    shared_correct_matched_imgs = Value('i', 0)
    shared_unknown_matched_imgs = Value('i', 0)
    shared_tot_imgs = Value('i', 0)
    counter = Value('i', 0)
    shared_lock = Lock()
    num_workers = cpu_count()
    
    feature = load_feature(modelspath)
    
    pool = Pool(initializer=init_globals,
                initargs=[counter, shared_tot_imgs, shared_correct_matched_imgs, shared_unknown_matched_imgs,
                          shared_lock], processes=num_workers)
    
    for img in glob.glob(os.path.join(imtodetect_path, '*g')):
        pool.apply_async(predict_single_img, (img,imtodetect_path,excel_file_path,files_path,use_top3,uid,matches_path,unknown_path,num_query_photos,index,modelspath))
        index+=increment
    
    pool.close()
    pool.join()

The array is created with the instruction feature = load_feature(modelspath). This is the array that I want to share.

In init_globals I inizialize the shared value:

def init_globals(counter, shared_tot_imgs, shared_correct_matched_imgs, shared_unknown_matched_imgs, shared_lock):
    global cnt, tot_imgs, correct_matched_imgs, unknown_matched_imgs, lock
    cnt = counter
    tot_imgs = shared_tot_imgs
    correct_matched_imgs = shared_correct_matched_imgs
    unknown_matched_imgs = shared_unknown_matched_imgs
    lock = shared_lock
Improve this question
9
  • 1
    Did you try multiprocessing.Array? Commented Apr 3, 2021 at 17:33
  • If you are on a mac or linux machine, you can set the start method to "fork" and create the data at the module level, and it will be copied across to the children automatically due to the copy-on-write memory used by fork. This only really works for read-only data because any changes don't get reflected in other children or the parent process. Commented Apr 3, 2021 at 17:56
  • You may not be modifying the array, but what about the objects within the array. What type of objects are they? Are they able to be pickled? Will their state be modified and so they must be shared? Commented Apr 3, 2021 at 18:00
  • Can you use Pool.map? Example: docs.python.org/3/library/multiprocessing.html Commented Apr 3, 2021 at 18:55
  • @MarkTolonen I read the documentation of multiprocessing.Array but I did not find anything about array of objects. Commented Apr 3, 2021 at 21:39

1 Answer 1

Reset to default
1

The easy way of providing shared static data is simply to make it a global variable accessible to the function you want to call. If you're using an operating system which supports "fork", it is very straightforward to use global variables in child processes as long as they're constant (if you modify them, changes won't be reflected in the other processes)

import multiprocessing as mp
from random import randint

shared = ['some', 'shared', 'data', f'{randint(0,1e6)}']

def foo():
    print(' '.join(shared))

if __name__ == "__main__":
    mp.set_start_method("fork")
    #defining "shared" here would be valid also
    p = mp.Process(target=foo)
    p.start()
    p.join()
    
    print(' '.join(shared)) #same random number means "shared" is same object

This won't work when using "spawn" as the start method (the only one available on windows), because the memory of the parent is not shared in any way with the child, so the child must "import" the main file to gain access to whatever the target function is (this is also why you can run into problems with decorators.) If you define your data outside the if __name__ == "__main__": block, it will kinda work, but you will have made separate copies of the data, which can be undesirable if it's big, slow to create, or can change each time it's created.

import multiprocessing as mp
from random import randint

shared = ['some', 'shared', 'data', f'{randint(0,1e6)}']

def foo():
    print(' '.join(shared))

if __name__ == "__main__":
    mp.set_start_method("spawn")

    p = mp.Process(target=foo)
    p.start()
    p.join()
    
    print(' '.join(shared)) #different number means different copy of "shared" (1 a million chance of being same i guess...)
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

But even with mp.set_start_method("spawn"), if you needed all processes to see the same data even when you had random generated data like you have, you would be smart enough to call at the global scope random.seed(1), for example, just before you define shared and then the main process and the sub-process would print identical results. It's possible to find cases where it will not work for spawn, but it may not be relevant to the OP's situation. You just need initialization that is repeatable.
I think your answer is also implying that with fork, unlike with spawn, memory is being shared between the main process and a sub-process, but that is not true. If it were, you would never have to pickle and unpickle arguments passed to to your sub-process. It's just that the main process's address space is copied to the new address space as it existed at the point of forking rather than starting off with an "empty" address space that get initialized by re-executing the global code. I think the problem is in the use of your phrase "shared with the child". It should be "copied to child".
@Booboo Yes, if the initialization is fully repeatable, spawn would still be acceptable. Random was just a quick example to demonstrate that the object was indeed re-created rather than copied. I somewhat disagree with your assessment about fork because until one of the processes modifies the given chunk of memory, it actually does point to the same physical place in memory for both processes. I think in that case "shared" is an appropriate enough word.
So you are saying that fork is “copy on write”. Interesting.
I wasn’t disagreeing. I did not know that and honestly found it interesting.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.