multiprocessing in python: AttributeError

Question

I'm trying to use multiprocessing, but I keep getting this error:

AttributeError: Can't get attribute 'processLine' on <module '__main__'

(The processLine function returns word, so I guess the problem is here, but I don't know how to get around it)

import multiprocessing as mp

pool = mp.Pool(4)
jobs = []
Types =[]

def processLine(line):
     line = line.split()
     word = line[0].strip()
     return word

with open("1.txt", "r", encoding = "utf-8") as f:
     for line in f:
          word = (jobs.append(pool.apply_async(processLine,(line))))
          Types.append(word)
     filtered_words=[]
     with open("2.txt", "r", encoding = "utf-8") as f:
          for line in f:
               word = jobs.append(pool.apply_async(processLine,(line)))
               if word in Types:
                    filtered_words = "".join(line)
     print(filtered_words)

for job in jobs:
    job.get()

pool.close()

And this is what I get: Process ForkPoolWorker-1:

Process ForkPoolWorker-2:

Process ForkPoolWorker-3:

Process ForkPoolWorker-4:

Traceback (most recent call last):

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap

self.run()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker

task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res)

AttributeError: Can't get attribute 'processLine' on

Traceback (most recent call last):

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap

self.run()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run

self._target(*self._args, **self._kwargs)

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker

task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get() File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res) File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap self.run() File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get return _ForkingPickler.loads(res) File "/Users/user/anaconda/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) AttributeError: Can't get attribute 'processLine' on AttributeError: Can't get attribute 'processLine' on File "/Users/user/anaconda/lib/python3.6/multiprocessing/pool.py", line 108, in worker task = get()

File "/Users/user/anaconda/lib/python3.6/multiprocessing/queues.py", line 345, in get

return _ForkingPickler.loads(res)

AttributeError: Can't get attribute 'processLine' on

If (line) needs to be a tuple, do (line,). A comma makes a tuple, not the parens. — Mad Physicist
– Mad Physicist, Commented Nov 29, 2017 at 2:47
Which actual line raises the error? Could you post the full stack trace? — Mad Physicist
– Mad Physicist, Commented Nov 29, 2017 at 3:16
Also, I am unable to reproduce your error by just running this in a Py3.6.2 interpreter... — Mad Physicist
– Mad Physicist, Commented Nov 29, 2017 at 3:19

Mad Physicist · Accepted Answer · 2017-11-29 17:11:32Z

4

The multiprocessing module needs to be able to import your module safely. Any code not inside a function or class should be protected by the standard Python import guard:

if __name__ == '__main__':
    ...code goes here...

But there are other problems with your code. For example, you've got:

word = jobs.append(pool.apply_async(processLine,(line)))

...but append doesn't return a value, so this will always assign None to word.

Rather than using a for loop to repeatedly call pool.apply_async, you may want to consider using pool.map_async instead, or just pool.map if you don't actually need the asynchronous behavior.

edited Nov 29, 2017 at 17:11

Mad Physicist

116k29 gold badges202 silver badges292 bronze badges

answered Nov 29, 2017 at 3:10

larsks

318k49 gold badges473 silver badges482 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Mad Physicist Over a year ago

Does this explain OP's issue?

larsks Over a year ago

Does it not? I mean, obviously I think it does or I wouldn't have bothered posting it here. I suspect that the unprotected code in the file is causing the error, but since it's not a mcve I can't actually try running it myself without futzing around with the code more than I want.

Mad Physicist Over a year ago

I dunno, was able to run the code fine with made up inputs. Didn't get OP's error or need the guard block.

Vince P. · Accepted Answer · 2020-08-03 22:27:16Z

0

I worked around the AttributeError issue by using VS Code in administrator mode to run it instead of Anaconda Spyder.

answered Aug 3, 2020 at 22:27

Vince P.

5094 silver badges5 bronze badges

Comments

Blue Robin · Accepted Answer · 2023-11-08 20:44:45Z

0

I faced the same issue. So, I put all functions in another .py file and imported ipnyb, and called them inside the main function.

edited Nov 8, 2023 at 20:44

Blue Robin

1,1536 gold badges17 silver badges40 bronze badges

answered Nov 4, 2023 at 18:15

Jaswanth Alla

1

Collectives™ on Stack Overflow

multiprocessing in python: AttributeError

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related