Can't use multiprocessing in python function

Question

I'm a beginner of python, I'm trying to put multiprocessing into a function, however python gives me an error.

Please refer the original code as below:

from multiprocessing import Process
import time
def func1():
    print('test1')
    time.sleep(10)  
def func2():
    print('test2')
    time.sleep(5)
if __name__ == '__main__':
    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()  
    print('done')

It runs well and give the correct result I need.

However, when I tried to put the multiprocessing code into function:

from multiprocessing import Process
import time

def test_multiprocessing():
    def func1():
        print('test1')
        time.sleep(10)  
    def func2():
        print('test2')
        time.sleep(5)
    if __name__ == '__main__':
        p_func1 = Process(target=func1)
        p_func2 = Process(target=func2)
        p_func1.start()
        p_func2.start()
        p_func1.join()
        p_func2.join()  
        print('done')

test_multiprocessing()

Below is error I got, may I know how to fix this issue ? The reason I'd like to put multiprocessing into a funciton is because there is an existing code there, and I don't want to do major change of the code to support multiprocessing.

Traceback (most recent call last):
  File "multipleprocessing.py", line 20, in <module>
    test_multiprocessing()
  File "multipleprocessing.py", line 14, in test_multiprocessing
    p_func1.start()
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'test_multiprocessing.<locals>.func1'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\spawn.py", line 99, in spawn_main
    new_handle = reduction.steal_handle(parent_pid, pipe_handle)
  File "C:\Users\User\AppData\Local\Programs\Python\Python36\lib\multiprocessing
\reduction.py", line 87, in steal_handle
    _winapi.DUPLICATE_SAME_ACCESS | _winapi.DUPLICATE_CLOSE_SOURCE)
PermissionError: [WinError 5] Access is denied

Per tested code on Linux, it works. Does that mean Windows Python can't support multiprocessing in function?

AttributeError: Can't pickle local object 'test_multiprocessing.<locals>.func1' make func1 and func2 module-level functions. — Stop harming Monica
– Stop harming Monica, Commented May 21, 2018 at 11:19

Prakash Palnati · Accepted Answer · 2018-05-21 09:32:48Z

3

Your code is correct. You shouldn't be keeping if __name__ == '__main__': inside the function. Read more about it here why name=="main"

try like below,

from multiprocessing import Process
import time

def test_multiprocessing():
    def func1():
        print('test1')
        time.sleep(10)  
    def func2():
        print('test2')
        time.sleep(5)

    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()  
    print('done')

test_multiprocessing()

answered May 21, 2018 at 9:32

Prakash Palnati

3,43927 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Knight Over a year ago

Hi, Prakash. Thanks for reply. I tried to remove if name == 'main':, however still got the same error there .

Prakash Palnati Over a year ago

copy paste the exact code which I answered. It works fine for me. Also take care of indentation

Knight Over a year ago

Tried to copy and paste your code, still got same error. I'm using Python 3.6.5, is there any version dependency ?

Prakash Palnati Over a year ago

should not be a problem. Looks like some access is being denied. If you are windows, try running the cmd as administrator and run your program

Knight Over a year ago

I tested your code on Linux, it works. But in Windows, it still has the same problem. So, Windows python can't support multiprocessing in a function?

|

Chanda Korat · Accepted Answer · 2018-05-25 03:53:27Z

1

a bit correction in @Prakash answer. You need to call function inside from if __name__== "__main__"

Here, explained well !!

from multiprocessing import Process
import time


def func1():
    print('test1')
    time.sleep(10)

def func2():
    print('test2')
    time.sleep(5)

def test_multiprocessing():
    p_func1 = Process(target=func1)
    p_func2 = Process(target=func2)
    p_func1.start()
    p_func2.start()
    p_func1.join()
    p_func2.join()
    print('done')

if __name__== "__main__":
    test_multiprocessing()

Another way is you can bound method to a class because functions are only picklable if they are defined at the top-level of a module. as below:

from multiprocessing import Process
import time

class Foo:

    def func1(self):
        print('test1')
        time.sleep(10)

    def func2(self):
        print('test2')
        time.sleep(5)

    def test_multiprocessing(self):
        p_func1 = Process(target=self.func1)
        p_func2 = Process(target=self.func2)
        p_func1.start()
        p_func2.start()
        p_func1.join()
        p_func2.join()
        print('done')

if __name__== "__main__":
    f=Foo()
    f.test_multiprocessing()

edited May 25, 2018 at 3:53

answered May 21, 2018 at 9:47

Chanda Korat

2,5812 gold badges22 silver badges24 bronze badges

2 Comments

Knight Over a year ago

Thanks for reply Chanda, it works on Windows. If I need to put func1, func2 into function test_multiprocessing, is that possible?

Chanda Korat Over a year ago

@Knight please see updated answer. Directly putting func1,func2 into functiontest_multiprocessing would give you pickling error. You can use class instead.

Collectives™ on Stack Overflow

Can't use multiprocessing in python function

2 Answers 2

6 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related