Python Multiprocessing Pool passing function arguments

Question

I've got 4 nested loops, where I loop though lists of lists, and a function where I pass some arguments I get from the function, specifically:

def method(var1, var2 ....., var 16):
  #computation 

pool = Pool()

for values1 in list_of_values1:
  #assign 4 variables from the list 
  for values2 in list_of_values2:
    #assign 4 variables from list 
    for values3 in list_of_values3:
      #assign 4 variables from  list 
      for values4 in list_of_values4:
        #assign 4 variables from list
        method(var1, ..., var16)

I have tried using the

pool.map(method, [var1,..., var16]) in order to paralelize the whole process but it throws an error saying "method() takes exactly 16 arguments, 1 given"

I have also tried to use Threads but it does not improve much.

Grateful for the help!

Cheers

can you show your attempted pool.map call?

roippi
– roippi

2014-05-06 18:11:04 +00:00
Commented May 6, 2014 at 18:11 — roippi
– roippi, Commented May 6, 2014 at 18:11
pool.map(method, [var1,..., var16]) like this

Claudiu S
– Claudiu S

2014-05-06 18:15:52 +00:00
Commented May 6, 2014 at 18:15 — Claudiu S
– Claudiu S, Commented May 6, 2014 at 18:15

dano · Accepted Answer · 2014-05-06 18:16:42Z

5

Pool.map is designed to call method on each value in the iterator you pass as the second argument. So in your example, it will try to do the following in parallel:

method(var1)
method(var2)
method(var3)
...
method(var16)

I think what you want to do is:

for values1 in list_of_values1:
  #assign 4 variables from the list 
  for values2 in list_of_values2:
    #assign 4 variables from list 
    for values3 in list_of_values3:
      #assign 4 variables from  list 
      for values4 in list_of_values4:
        #assign 4 variables from list
        pool.apply_async(method, (var1, ..., var16))
pool.close()
pool.join()

This is just calling method with the 16 arguments it expects, but doing it in a background process.

It's also possible you can do something a little nicer than the multiple nested for loops, by using itertools to build a list of (var1, ..., var16) tuples, which could then be passed to pool.map. But it depends on what #assign 4 variables from list* actually does.

edited May 6, 2014 at 18:16

answered May 6, 2014 at 18:11

dano

95.5k21 gold badges234 silver badges231 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Claudiu S Over a year ago

If it tries to do method(var1) method(var2) method(var3) ... method(var16) I'm not really sure how that helps, because I need all of the values when I make the function call. Maybe I am not understanding this properly

dano Over a year ago

Exactly. I was trying to explain why pool.map the way you're using it isn't helpful. The code example I included that uses pool.apply_async does do the right thing, though.

Claudiu S Over a year ago

Right, cheers, I'll try it off now :) I need the for loops. **It's how the algorithm works

Collectives™ on Stack Overflow

Python Multiprocessing Pool passing function arguments

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related