Utilizing the multi processing tool with Numpy Python

Question

I am trying to utilize multi processing tool with numpy arrays. I want to run func1() and func2() simultaneously and then use np.concatenate() and place Solution_1 and Solution_2 together. How will I be able to do such a thing?

Modules:

import multiprocessing
import numpy as np

Code:

Numbers = np.array([1,2,3,4,5,6,7,8,9,10,11,12])
def func1():
     Solution_1 = Numbers + 10 
     return Solution_1 
def func2():
     Solution_2 = Numbers * 10
     return Solution_2 
#setting up the multi processing vars     
p1 = multiprocessing.Process(target=func1)
p2 = multiprocessing.Process(target=func2)

#running the multi processes 
if __name__ == "__main__":
    p1.start()
    p2.start()

Rafael Valero · Accepted Answer · 2021-02-09 07:53:45Z

1

For example:

import multiprocessing as mp
import numpy as np
num_cores = mp.cpu_count()

Numbers = np.array([1,2,3,4,5,6,7,8,9,10,11,12])
def func1():
     Solution_1 = Numbers + 10
     return Solution_1
def func2():
     Solution_2 = Numbers * 10
     return Solution_2

# Getting ready my cores, I left one aside
pool = mp.Pool(num_cores-1)
# This is to use all functions easily
functions = [func1, func2]
# This is to store the results
solutions = []
for function in functions:
    solutions.append(pool.apply(function, ()))

solutions looks like:

[array([11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22]),
 array([ 10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110, 120])]

You could place that directly in a dataframe:

solutions = pd.DataFrame(solutions)
print(solutions)
    0   1   2   3   4   5   6   7   8    9   10   11
0  11  12  13  14  15  16  17  18  19   20   21   22
1  10  20  30  40  50  60  70  80  90  100  110  120

I see in a similar cases / alternatives:

Global Variables, this is very similar to the original case post here: https://stackoverflow.com/a/40399562/7127519
apply_async: https://stackoverflow.com/a/48933782/7127519
A shared variable to communicate: How can I recover the return value of a function passed to multiprocessing.Process?
multiprocessing.Manager(): https://stackoverflow.com/a/10415215/7127519

And others.

edited Feb 9, 2021 at 7:53

answered Feb 9, 2021 at 0:07

Rafael Valero

2,83623 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

tony selcuk Over a year ago

When I run the code I get an error saying AttributeError: 'dict' object has no attribute 'append in the solutions.append(pool.apply(function, ())) bit of the code.

Rafael Valero Over a year ago

Sorry, that was because I defined solutions as dict instead of a list. Solve now.

tony selcuk Over a year ago

For some reason it runs forever and does not stop

Rafael Valero Over a year ago

Where does it get stack? in the for?

tony selcuk Over a year ago

I will upload a snapshot to the issue perhaps that would help

|

Collectives™ on Stack Overflow

Utilizing the multi processing tool with Numpy Python

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related