Python - How can I pass multiple arguments using Pool map [duplicate]

Question

This is a sniplet of my code:

data = [currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile]
results = multiprocessing.Pool(5).map(partial(self.postAd,data),range(3))
...
def postAd (self,login,password,titlesFile,licLocFile,subCity,bodiesMainFile,bodiesKeywordsFile,bodiesIntroFile):
...

(Just so you know what's going on: currentAccount and campaign are classes, those are variables within those classes. Using self b/c this is all being run in a class. I'm trying to run self.postAd 3x passing it all the variables I have in data)

When I run that it says " postAd() missing 6 required positional arguments: 'titlesFile', 'licLocFile', 'subCity', 'bodiesMainFile', 'bodiesKeywordsFile', and 'bodiesIntroFile'"

What am I doing wrong? Why does it only accept 2 variables?

If I cant use Pool map, how should I be doing this?

I also tried this with no success:

results = multiprocessing.Pool(5).map(lambda args: self.postAd(currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile), range(3))

Error: Can't pickle <function NewPostService.processNewAds.<locals>.<lambda> at 0x0000000002F3CBF8>: attribute lookup <lambda> on functions failed

I also tried this with no success: results = multiprocessing.Pool(5).map(lambda args: self.postAd(currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile), range(3)) Error: Can't pickle <function NewPostService.processNewAds.<locals>.<lambda> at 0x0000000002F3CBF8>: attribute lookup <lambda> on functions failed — Daniel Rusu
– Daniel Rusu, Commented Sep 28, 2016 at 23:22
Do you really want all three calls to use exactly the same arguments? I'm not sure that works with map, since it will always want to be providing an extra argument from the sequence it loops over (the range in this case). You can probably make it work by writing an extra function that ignores its last argument (and using partial(self.postadd, *data)), but it might be easier to use your own loop creating Processes. — Blckknght
– Blckknght, Commented Sep 28, 2016 at 23:41
Yes I want it to use the same arguments. I want the same process to happen 3x simultaneously is the reason why. How would I do this " but it might be easier to use your own loop creating Processes." Can you give me an example based on my code if you'd be so kind — Daniel Rusu
– Daniel Rusu, Commented Sep 29, 2016 at 0:26

Alex Hall · Accepted Answer · 2016-09-29 10:55:36Z

0

Your first attempt is a misuse of partial. data is a single argument: it being a list doesn't automatically unpack its contents. partial simply takes variable arguments and so you should pass those arguments 'normally', either

partial(self.postAd, currentAccount.login,currentAccount.password, ...

or

partial(self.postAd, *data)

The reason it says that postAd received two arguments instead of just one (data) is that it also implicitly received the self argument.

Your second attempt is the right idea and very close, but it so happens that in older versions of python pickle (which is essential for multiprocessing) can't handle lambdas. Replace the lambda with a named function defined using def:

def postNow(arg):  # Accepts an argument from map which it ignores
    self.postAd(currentAccount.login, ..., campaign.bodiesIntroFile)

Side notes:

It's a bit odd that your argument to the lambda is called args when it's just one argument. If your intention is to make it flexible and accept variable arguments, use lambda *args: ... or even lambda *args, **kwargs: ... to accept keyword arguments. The names are not important, they are just convention. The important part is the * and **. Note that partial has a signature like this, as an example.

I have never seen this issue with the second attempt before. I learnt it by Googling the error message. Always Google things.

edited Sep 29, 2016 at 10:55

answered Sep 29, 2016 at 0:43

Alex Hall

36.2k5 gold badges63 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Daniel Rusu Over a year ago

Thanks for the response Alex. I'm still a beginner python programmer so some of this is over my head. I tried doing both options of 1. partial(self.postAd, *data) and 2. multiprocessing.Pool(5).map(partial(self.postAd,currentAccount.login,currentAccount.password,campaign.titlesFile,campaign.licLocFile,campaign.subCity,campaign.bodiesMainFile,campaign.bodiesKeywordsFile,campaign.bodiesIntroFile),range(3) ----------------- . and got this error for both of them: Error: postAd() takes 9 positional arguments but 10 were given. I'm not sure where the 10th argument is as I'm only passing 8.

Alex Hall Over a year ago

@DanielRusu it's getting the 8 normal arguments, self, and the 10th is the value from range(3) passed in by map. It wouldn't make much sense to modify postAd for this case so actually just forget about trying to use partial, it's not intended for this.

Collectives™ on Stack Overflow

Python - How can I pass multiple arguments using Pool map [duplicate]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related