Multiprocessing working in Python but not in iPython

Question

I am running the following code in iPython:

import multiprocessing

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x


if __name__ == '__main__':
    pool = multiprocessing.Pool()
    results = pool.map(my_function, [1,2,3,4,5,6])
    print(results)

in ipython QT console on Windows. However, the code does not work -- the QT console just freezes up. The issue is specific to iPython (the code above should work for the regular Python 2.7).

Any solution to this?

Works fine on Ubuntu. Must be an issue with windows. Do you paste the code into Ipython? — Padraic Cunningham
– Padraic Cunningham, Commented May 13, 2014 at 21:09
Yes, I just tried it in Spyder IDLE. It works there. Any idea how to make it work in the actual QT console? — user3438258
– user3438258, Commented May 15, 2014 at 12:41
I second @PadraicCunningham's comment. Works fine on Linux distributions (I'm using Debian). — lostsoul29
– lostsoul29, Commented Sep 1, 2016 at 21:31

dano · Accepted Answer · 2014-05-13 21:05:33Z

24

From the documentation:

Note

Functionality within this package requires that the __main__ module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.Pool examples will not work in the interactive interpreter

answered May 13, 2014 at 21:05

dano

95.5k21 gold badges234 silver badges231 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user3438258 Over a year ago

Did you copy/paste the code in the QT console, or did someting else?

Armando Contestabile Over a year ago

The FINAL answer. No more effort struggling trying to get multiprocessing.Pool working on Jupyter notebook.

Hugues Over a year ago

The answer by @Maverick does work in both Jupyter Notebook and Jupyter Lab, for both 'fork' and 'spawn' multiprocessing.

Maverick Meerkat · Accepted Answer · 2020-01-02 20:42:00Z

21

At least in the current version of Jupyter Notebook (the successor of IPython) you can solve this by moving the target function to a separate module, and importing it.

I have no idea why this works, it's rather odd, but it does.

i.e. - in a workers.py file put

def my_function(x):
    """The function you want to compute in parallel."""
    x += 1
    return x

Then in IPython/Jupyter notebook put:

import multiprocessing
import workers

pool = multiprocessing.Pool()
results = pool.map(workers.my_function, [1,2,3,4,5,6])
print(results)

also - the if-main thingy doesn't seem to be needed.

Credit: Gaurav Singhal

answered Jan 2, 2020 at 20:42

Maverick Meerkat

6,4723 gold badges54 silver badges71 bronze badges

4 Comments

John Zhang Over a year ago

This works but it seems like the performance has no increase because it looks like still running in a single process/thread.

Maverick Meerkat Over a year ago

@JohnZhang I get different process id's when I run this (although sometimes I do get the same process). You can just add os.getpid() to the worker.py (make sure to import os).

John Zhang Over a year ago

In my test, I will get MP if I run it in Python console. However, if I run it in Jupyter, it will be always 1 process. I think it makes sense since one Jupyter notebook is one process in Jupyter architecture (at least on Windows).

Maverick Meerkat Over a year ago

@JohnZhang have you used pool? I also use windows and I do get multiprocessing. Also it doesn't make sense, as every IDE or even console you use is one process, and they all spawn different processes.

Collectives™ on Stack Overflow

Multiprocessing working in Python but not in iPython

2 Answers 2

3 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related