Python Using List/Multiple Arguments in Pool Map

Question

I am trying to pass a list as a parameter to the pool.map(co_refresh, input_list). However, pool.map didn't trigger the function co_refresh. And also no error returned. It looks like the process hung in there.

Original Code:

from multiprocessing import Pool
import pandas as pd
import os

account='xxx'
password='xxx'
threads=5
co_links='file.csv'

input_list=[]

pool = Pool(processes=threads)
def co_refresh(url, account, password, outputfile):

    print(url + ' : ' + account + ' : ' + password + ' : ' + outputfile)

    return;

link_pool = pd.read_csv(co_links, skipinitialspace = True)

for i, row in link_pool.iterrows():

    ln = (row.URL, account, password, os.path.join('e:/', row.File_Name.split('.')[0] + '.csv'))

    input_list.append(ln)

pool.map(co_refresh, input_list)

pool.close()

However, it never triggered the function co_refresh. How can I use the list as a parameter to be passed to my function?

Old Question (Simplified):

I have below input_list, which is a list of list:

[a1, b1, c1, d1]
[a2, b2, c2, d2]
[a3, b3, c3, d3]

I have the function as below:

def func(a, b, c, d)
   ###
    return;

I would like to use multiprocess for this function func:

from multiprocessing import Pool
pool = Pool(processes=5)
pool.map(func, input_list)
pool.close()

However, it never triggered the function func. How can I use the list as a parameter to be passed to my function?

This is not exactly the code you tested, is it? It is full of errors. Post exactly what you tried and then you'll get an exact answer. — zvone
– zvone, Commented Nov 21, 2017 at 23:50
As @zvone suggested, you should share with us the exact code you have tested and the error reported by Python interpreter. Currently the pieces you give doesn't compose a runnable script. — Patrick the Cat
– Patrick the Cat, Commented Nov 21, 2017 at 23:54
On the top level of the script, you should only define functions and classes, not actally execute any of it. The code to be executed should be called from withing a if __name__ == "__main__" block. That is in general a good idea, but in case of multiprocessing, it is essential. Multiprocessing will import your module again in another process. The way it is, that will cause all of the coe to be executed again, rather than what you'd expect. — zvone
– zvone, Commented Nov 22, 2017 at 22:48

georgexsh · Accepted Answer · 2017-11-27 22:05:57Z

17

+25

You should define your work function before declaring the Pool, when you declaring Pool, sub worker processes forked from that point, worker process don't execute code beyond that line, therefore not seeing your work function.

Besides, you'd better replace pool.map with pool.starmap to fit your input.

A simplified example:

from multiprocessing import Pool

def co_refresh(a, b, c, d):
    print(a, b, c, d)

input_list = [f'a{i} b{i} c{i} d{i}'.split() for i in range(4)]
# [['a0', 'b0', 'c0', 'd0'], ['a1', 'b1', 'c1', 'd1'], ['a2', 'b2', 'c2', 'd2'], ['a3', 'b3', 'c3', 'd3']]

pool = Pool(processes=3)
pool.starmap(co_refresh, input_list)
pool.close()

edited Nov 27, 2017 at 22:05

answered Nov 27, 2017 at 8:44

georgexsh

16.8k3 gold badges45 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lovechillcool Over a year ago

I finally got time to test this... I now understand where the issue is... However, the process completed but didn't print anything. Do you know why is that?

georgexsh Over a year ago

@lovechillcool are you using the code I posted? note that it is for python3.6.

Tarun Lalwani · Accepted Answer · 2017-11-27 08:49:58Z

Consider the below code

from multiprocessing.pool import Pool

data = [["a1", "b1", "c1", "d1"],
        ["a2", "b2", "c2", "d2"],
        ["a3", "b3", "c3", "d3"], ]


def someaction(a, b=1, c=2, d=3):
    print(a, b, c, d)

When you call this in your script using a pool

pool = Pool(4)
pool.map(someaction, data)

The output is

['a1', 'b1', 'c1', 'd1'] 1 2 3
['a2', 'b2', 'c2', 'd2'] 1 2 3
['a3', 'b3', 'c3', 'd3'] 1 2 3

So a gets the array and rest all parameters are not passed. Pool.map expects a function to only have one argument. So for your case to work you need to create a wrapper function

def someaction_wrapper(data):
    someaction(*data)

And then call this wrapper function in pool. Now you use

pool = Pool(4)
pool.map(someaction_wrapper, data)

And the output is

a1 b1 c1 d1
a2 b2 c2 d2
a3 b3 c3 d3

Which is what you wanted I believe

lovechillcool · Accepted Answer · 2017-12-18 06:47:31Z

0

georgexsh's answer works perfectly in Python 3; the key is that starmap allows to pass multiple arguments into the function.

However, if you use Python 2, you will need to use python classical unpacking mentioned in comments by Ahmed under the question here.

In my case, I just need to "enlist" the argument first in the function.

def func(args)
   (a, b, c, d) = args
   # You can then use a, b, c, d in your function
    return;

edited Dec 18, 2017 at 6:47

answered Dec 17, 2017 at 20:48

lovechillcool

7822 gold badges13 silver badges33 bronze badges

Collectives™ on Stack Overflow

Python Using List/Multiple Arguments in Pool Map

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related