Using python multiprocess pool to a function

Question

I have a function

def dist_to_center(ra_center,dec_center):
    # finding theta
    cos_ra = np.cos(ra_center-var1['ra'])
    cos_dec = np.cos(dec_center-var1['dec'])
    sin_dec = np.sin(dec_center)*np.sin(var1['dec'])

    theta = np.arccos((cos_ra*cos_dec)+sin_dec*(1-cos_ra))
    numerator = theta*comoving_dist
    denominator = 1+var1['zcosmo']

    # THE FINAL CALCULATED DISTANCE TO CENTRE
    dist_to_center = (numerator/denominator) 
    return dist_to_center

I want to make use of my processors, so I am using multiprocess pool like this:

if __name__ == '__main__':
    pool = Pool(processes=6)
    pool.map(dist_to_center, ra_center, dec_center) #calling the function with it's inputs
    pool.close()
    pool.join()

The code seems to be proper and is working, but only 1 processor is running instead of the 6 I have called. What am I doing wrong here?

You're saying that ra_center and dec_center are one-dimensional arrays when you call pool.map()? — John Zwinck
– John Zwinck, Commented Oct 9, 2014 at 12:01
@JohnZwinck ra_center and dec_center are numpy array's which are inputs for my function. Should I not call them in pool.map?? Because I thought pool.map takes the function and then it's inputs — user3397243
– user3397243, Commented Oct 9, 2014 at 12:03
Sure, but you're telling us you have only a single pair of arrays. So you would need to slice the arrays into multiple parts to calculate in parallel--the Pool has no idea how to divide your 440K element arrays into subparts properly. — John Zwinck
– John Zwinck, Commented Oct 9, 2014 at 12:05

John Zwinck · Accepted Answer · 2014-10-09 12:13:12Z

1

You are passing a pair of one-dimensional arrays to the Pool. You need to slice the arrays yourself to make the Pool understand how to process them efficiently. For example:

def dist_to_center_mapper(arrays):
    return dist_to_center(arrays[0], arrays[1])

ra = np.split(ra_center, 6)
dec = np.split(dec_center, 6)
pool = Pool(processes=6)
pool.map(dist_to_center_mapper, zip(ra, dec))

I think the "mapper" function is required because Pool.map() takes only a single iterable of arguments. So we zip together the two lists of array slices so they get doled out together to the multiple processes. Note that you could split the arrays into more pieces than the number of processes if you want, if some pieces may take different amounts of time etc.

answered Oct 9, 2014 at 12:13

John Zwinck

252k44 gold badges346 silver badges459 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user3397243 Over a year ago

So I just need to create a new function called dist_to_center_mapper and call that function in pool.map to make it work?

John Zwinck Over a year ago

Something like that, because I think Pool.map wants to call a function taking a single argument. You could also edit your existing function to take a single argument. The argument it receives will be a tuple containing two sub-arrays, so you can see why my "mapper" function does what it does.

user3397243 Over a year ago

I tried the above by splitting the arrays, but still only one processor seems to be running! I would like to mention that all the arrays inside my function are numpy arrays with shape 443726. My function just does some math to give me a distance. Can I just say def dist_to_center()

John Zwinck Over a year ago

Why don't you add a print statement to the top of dist_to_center() and see if it gets called in parallel or not.

user3397243 Over a year ago

I have opened a chat room, would it be possible for you to join, I can explain exactly what I am doing in the functino

Collectives™ on Stack Overflow

Using python multiprocess pool to a function

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related