Parallelize a function with data-frame as input

Question

How I can parallelize the func1 in below code. It takes data-frame as input. func1 is the prototype of the function I am trying to parallelize. In real function have many data frames and series as input in the function.

import pandas as pd

def func1(d,a):
    product = d.prod(axis=1)*a
    print(product)

#input 1
a1=2    
input1 = pd.DataFrame({'F':[2,3,4,5,1,2], 'E':[12,4,5,6,7,2], 'N':[2,7,6,5,4,3]}) 

#input 2
a2 = 0
input2 = pd.DataFrame({'F':[0,32,4,12,1,2], 'E':[1,4,5,0,7,2], 'N':[21,7,61,5,4,3]})

#input3
a3=100
input3 = pd.DataFrame({'F':[0,1,1,1,1,1], 'E':[1,12,5,110,7,2], 'N':[3,7,61,5,1,1]})

#call function
func1(input1,a1)
func1(input2,a2)
func1(input3,a3)

Renard Korzeniowski · Accepted Answer · 2018-12-12 21:47:55Z

2

Just use built-in library multiprocessing. I've edited your code to work on multiple cores simultaneously.

import pandas as pd
from multiprocessing import Pool, cpu_count

CORE_NUMBER = cpu_count()

def func1(d, a):
    product = d.prod(axis=1)*a
    print(product)

# input 1
a1=2    
input1 = pd.DataFrame({'F':[2,3,4,5,1,2], 'E':[12,4,5,6,7,2], 'N':[2,7,6,5,4,3]}) 

# input 2
a2 = 0
input2 = pd.DataFrame({'F':[0,32,4,12,1,2], 'E':[1,4,5,0,7,2], 'N':[21,7,61,5,4,3]})

# input 3
a3=100
input3 = pd.DataFrame({'F':[0,1,1,1,1,1], 'E':[1,12,5,110,7,2], 'N':[3,7,61,5,1,1]})

data = [(input1,a1),(input2,a2),(input3,a3)]
pool = Pool(CORE_NUMBER) 

# call function
pool.starmap(func1, data)
pool.close()
pool.join()

edited Dec 12, 2018 at 21:47

answered Dec 12, 2018 at 21:37

Renard Korzeniowski

1,1058 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Parallelize a function with data-frame as input

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related