Python Multiprocess Error

As the error suggests, you can't pickle instance methods. The problem is this line:

pool.map(job.do_me, ((x[i], y[i]),)) for i in range(len(x))

The mechanism behind this is that when the map function sends the function (the first argument) to all of the workers, it has to serialize it to data somehow, so it's using a mechanism called pickling. There are other mechanisms, this is the a very common one in Python).

When it's trying to pickle an instance method (specifically method do_me, of instances of type Worker) and send it to the pool (for the workers to know what method they're supposed to execute), it fails. Because you can't pickle instance methods.

You can fix this by moving the method to the module level (removing the Worker class):

def do_me(test_instance):
    return test_instance.add(x,y)

Now we don't have access to self, since we're using the test_instance that's sent here explicitly, so this method isn't bound to the Test class anymore... Or put in other words - this isn't an instance method anymore. Now make sure you re-factor everything to work as you plan.

The Test class should have something along this structure to keep the list comprehension in the argument construction to pool.map simple:

class Test():
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def add(self,x,y):
        return self.x + self.y

Adn then calling it like this:

results = pool.map(do_me, [Test(x[i], y[i]) for i in range(len(x))])

Full code:

import multiprocessing


class Test():
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def add(self):
        return self.x + self.y


def do_me(test_instance):

    return test_instance.add()


if __name__ == '__main__':

    x = [1,2,3]
    y = [9,7,5]

    pool = multiprocessing.Pool(processes=4)
    results = pool.map(do_me, [Test(x[i], y[i]) for i in range(len(x))])
    print results

Some notes:

1. pool.map already returns a list

2. __init__ is the standard place to initialize the object data (x, y in your case).

3. The function pool.map is using is applied to each item of the iterable, so it should be a single-argument function (you can use a tuple, but you have to unpack it).