python multiprocessing does not run functions

Question

I just want to see a simple code implementation of multiprocessing on windows, but it doesn't enter/run functions neither in jupyternotebook or running saved .py

import time
import multiprocessing
s=[1,4]
def subu(remo):
    s[remo-1]=remo*9
    print(f'here{remo}')
    return
if __name__=="__main__":
    p1=multiprocessing.Process(target=subu , args=[1])
    p2=multiprocessing.Process(target=subu , args=[2])
    p1.start()
    p2.start()
    p1.join()
    p2.join()
#     print("2222here")
print(s)
input()

the output by .py is:

[1, 4]
[1, 4]

and the output by jupyternotebook is:

[1,4]

which I hoped to be:

here1
here2
[9,18]

what's wrong with code above? and what about this code:

import concurrent
thread_num=2
s=[1,4]
def subu(remo):
    s[remo-1]=remo*9
    print(f'here{remo}')
    return
with concurrent.futures.ProcessPoolExecutor() as executor:
## or if __name__=="__main__":
##...    with concurrent.futures.ProcessPoolExecutor() as executor:
    results=[executor.submit(subu,i) for i in range(thread_num)]
    for f in concurrent.futures.as_completed(results):
        print(f.result())
input()

doesnot run at all in jupyter pulling error

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I kinda know I can't expect jupyter to run multiprocessing. but saved.py also can't run it. and it exits without waiting for input()

Running on command line in linux I got the hoped for result. Could not reproduce. — tdelaney
– tdelaney, Commented Aug 18, 2020 at 1:43
Okay, edit that, not quite the hoped for. here1\nhere2\n[1,4]. The [9,18] wouldn't happen because the subprocess memory (either copy-on-write or an independent process) isn't seen by the parent. — tdelaney
– tdelaney, Commented Aug 18, 2020 at 1:45
@tdelaney so how to apply changes on parent with multiprocessing, and what about 'here1' and 'here2', why they don't happen? — Farhang Amaji
– Farhang Amaji, Commented Aug 18, 2020 at 1:48
I'm not a regular jupyter user but just poking around saw this question suggesting that the worker be placed in a separate module so that it can be imported by the subprocess. — tdelaney
– tdelaney, Commented Aug 18, 2020 at 1:50
As for changes back in the parent, you'll need some way to communicate from process back to parent. multiprocessing.queue() can be that channel. And multiprocessing.Pool can also. Since I don't know the jupyter part, i'm not sure about answering. — tdelaney
– tdelaney, Commented Aug 18, 2020 at 1:52

Farhang Amaji · Accepted Answer · 2020-08-18 02:23:46Z

1

There are a couple of potential problems. The worker function needs to be importable (at least on Windows) so that it can be found by the subprocess. And since subprocess memory isn't visible to the parent, the results need to be returned. So, putting the worker in a separate module

subumodule.py

def subu(remo):
    remo = remo*9
    print(f'here{remo}')
    return remo

And using a process pool's existing infrastructure to return a worker return value to the parent. You could

import time
import multiprocessing
if __name__=="__main__":
    with multiprocessing.Pool(2) as pool:
        s = list(pool.map(subu, (1,2))) #here
    print(s)
    input()

edited Aug 18, 2020 at 2:23

Farhang Amaji

1,02117 silver badges33 bronze badges

answered Aug 18, 2020 at 1:58

tdelaney

77.9k6 gold badges91 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Farhang Amaji Over a year ago

I got ur point in how to treat with s list, and do u mean I can't get "here1' and 'here2' unless I treat them like an output of function?

tdelaney Over a year ago

I was worried about that. disappointing. Are you on Windows? Just searching "jupyter notebook multiprocessing windows" in that case is my only suggestion.

tdelaney Over a year ago

You need to get the calculation from remu() back to the parent. You could create a multiprocessing.queue() then create a multiprocessing.Process() and hand the queue down with the call. Pool does that part for you. The return value of the worker function is pickled in the process, returned to the parent, unpickled, and then iterated back from the map call. list just turns that return value iteration into a list. I noticed in your example that initial values in s didn't matter. Of course that's just a toy example and you may need to do something different.

Farhang Amaji Over a year ago

I didn't get the queue part, do I need .queue() for a purpose like, this example of getting handling a separate list. isn't the map enough?

tdelaney Over a year ago

You wouldn't use a queue if you use map.

|

Collectives™ on Stack Overflow

python multiprocessing does not run functions

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related