I have three scripts, scheduler.py which is a parallel task runner based on multiprocessing.Process and multiprocessing.Pipe, and the second script is simulation.pyx which is a script containing some classes and functions that I want to perform in parallel via scheduler.py and lastly a small main script where I create an instance of the parallelization class from scheduler.py, pass it to the classes in simulation.pyx and run the whole thing.

When the target parallel function is on the top level in simulation.pyx everything works fine, but as soon as I try to use scheduler.py with a class function in simulation.pyx I get a pickling error.

Since the code is several thousand of lines I'll only give some conceptual code:

small_main_script.py:

import simulation
import scheduler


if __name__ == '__main__':

    main = simulation.Main()
    scheduler = scheduler.parallel()
    main.simulate(scheduler)

simulation.pyx:

import scheduler

cdef do_something_with_job(job):
...

cdef class Main:
    cdef public ...
    ...

    def __init__(self):
    ...

    def some_function(self,job):
        ...
        do_something_with_job(job)
        ...

    def simulate(self, scheduler):

        for job in job_list:
            scheduler.add_jobs(job)

        scheduler.target_function = self.some_function

        scheduler.run_in_parallel()

The thing is that if I use useless dummy function like

def sleep(job):
    time.sleep(2)

and put it on the top level i.e. outside the classes, the parallelization works fine but as soon as i put it inside the class Main i get a pickling error. I get the same error if I use my real target function which is also defined in the class Main and I don't want to move it to the top level. The following is what happens when I use the dummy function sleep(self,job) inside the class Main. When it's outside the class it works fine.

PicklingError: Can't pickle <built-in method sleep of simulation.Main
object at 0x0D4A3C00>: it's not found as __main__.sleep

In [2]: Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1384, in load return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 864, in load dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 886, in load_eof

    raise EOFError
EOFError

I'm using Python 2.7

Update

I have managed to further isolate the problem. When using third party package pathos multiprocessing I'm able to pickle class functions. The problem now seems to be that I get an error when using function arguments that are class instances.