Pickle error when making parallel version of map function in Python using Multiprocessing

Question

I'm following the answer on Is there a simple process-based parallel map for python? using Python 3.6.1. I'm trying to use it the same exact way as I would use the regular map function. Why does this happen? How can I create a parallel map function for use with lambda?

def map_parallel(func, iterable, n_jobs=-1):
    if n_jobs == -1:
        n_jobs = multiprocessing.cpu_count()
    pool = multiprocessing.Pool(processes=n_jobs)
    return pool.map(func, iterable)

map_parallel(lambda x: x + 3, range(100))


PicklingError: Can't pickle <function <lambda> at 0x185284f28>: attribute lookup <lambda> on __main__ failed

falsetru · Accepted Answer · 2017-05-04 01:50:50Z

0

multiprocessing pickle objects to pass between processes. lambda is not serializable (See What can be pickled and unpickled?); it can't be passed between processes.

You need to define a function, and pass that.

def add3(n):
    return n + 3

map_parallel(add3, range(100))

Or for this case, you can use functools.partial with operator.add:

import functools
import operator

map_parallel(functools.partial(operator.add, 3), range(100))

answered May 4, 2017 at 1:50

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

O.rka Over a year ago

Is it possible to convert lambda to a function? I rarely ever use map with functions and use it on the fly with transforming iterables

falsetru Over a year ago

@O.rka, Check this answer.

Collectives™ on Stack Overflow

Pickle error when making parallel version of map function in Python using Multiprocessing

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related