how to parallelize a loop in python with a function with several arguments?

Question

I am trying to parallelize a for loop in python that must execute a function with several arguments, one of which will be changing through the loop. The loop itself needs to be embedded in a function. I have already looked here, here and here in stackoverflow and beyond (here and here) but I just cannot make it work :(

Below is a MWE:

import time
import numpy as np
from multiprocessing import Pool
from functools import partial

def mytestFun(otherStuff, myparams):
    return myparams[0]*otherStuff - myparams[1]

def myfun1(extraParams, mylist):
    [myMat, otherStuff] = extraParams
    
    for ivals in mylist:
        myparams = myMat[ivals,:]
        result = mytestFun(otherStuff, myparams)
    return result

if __name__ == '__main__':
    a_list = [0, 1, 2, 3, 4, 5]

    myMat = np.random.uniform(0,1,(6,2))
    extraParams = [myMat, 5]
    print(myfun1(extraParams, a_list))
    pool = Pool()
    func = partial(myfun1, extraParams)
    pool.map(func, a_list)
    pool.close()
    pool.join()

And I keep getting errors that I don't know how to interpret:

Traceback (most recent call last):
  File "exampleMultiProcessing.py", line 61, in <module>
    pool.map(func, a_list)
  File "/Users/laurama/miniconda3/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Users/laurama/miniconda3/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
TypeError: cannot unpack non-iterable int object

Thanks in advance!

Here dask.delayed can be your friend

rpanai
– rpanai

2020-06-25 23:47:23 +00:00
Commented Jun 25, 2020 at 23:47 — rpanai
– rpanai, Commented Jun 25, 2020 at 23:47
@ laura We can use joblib here;

Aditya
– Aditya

2020-06-26 00:22:14 +00:00
Commented Jun 26, 2020 at 0:22 — Aditya
– Aditya, Commented Jun 26, 2020 at 0:22

Aditya · Accepted Answer · 2020-06-26 18:30:18Z

3

You can read about joblib here. Basically when we use joblib, it expects that we will be passing it the args for the function which we want to parallelise. So here i am passing the args directly to the function, that's why i am looping using the underscore_variable, you can use anything there, no issues at all. Basically I am ignoring the looping variable using the _;

And yes, Parallel automatically will distribute it over n_cores;

Try this:

from joblib import Parallel, delayed    

if __name__ == '__main__': 
    a_list = [0, 1, 2, 3, 4, 5] 
    myMat = np.random.uniform(0,1,(6,2)) 
    extraParams = [myMat, 5] 
    print(myfun1(extraParams, a_list)) 
    result = Parallel(n_jobs=8)(delayed(myfun1)(extraParams, a_list) for _ in range(1))[0]

edited Jun 26, 2020 at 18:30

answered Jun 26, 2020 at 0:02

Aditya

2,5802 gold badges18 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Laura Over a year ago

it does thank you, but could you please provide more context? I don't understand very much what the solution is doing, in particular, this part: for _ in range(1)) in fact, I have never seen an underscore for a variable. Also, would Parallel automatically distribute the loop among the n_jobs? Thanks!

Aditya Over a year ago

@Laura; Apologies, i have updated the answer now; Let me know!

Laura Over a year ago

Actually, I don't understand what happens, but a quick check finds that this solution is actually significantly slower than just doing the loop both with this trivial example and with my real example... Any ideas?

Aditya Over a year ago

Ahh now I see why it's slower; you are having a loop over input's, I didn't see that. In that case we need to change the looping strategy in; try modifying it to loop over that array of yours(IE. On your inputs)

Collectives™ on Stack Overflow

how to parallelize a loop in python with a function with several arguments?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related