python multiple thread performance

Question

I have one server with 10 cpu cores, when I ran the following code which one thread, one of the cpu cores usage was 100%:

def fun():
    while 1:
        pass

but when I use 5 threads to run the same code, there were 5 cpu cores usages were: 30%, 20%, 15%, 25%, 12% respectively.

Why were not 5 cpu cores with 100% usages respectively? If multiple python threads can not fully use CPUs, how should multiple thread is faster than one thread?

This is probably because of the Global Interpreter Lock. Depending on your problem, you might be able to get around it by using multiprocessing. — mgilson
– mgilson, Commented Sep 6, 2015 at 1:45

HelloWorld · Accepted Answer · 2015-09-06 17:33:23Z

4

What you described is the typical bottleneck, which becomes more present when more cores try to get access to a single lock. As mgilson mentioned, this is the side-effect of the GIL which exists in many implementations of Python. The GIL in Python guarantees that only one thread executes Python byte-code at a time. This is for historical reasons. But as Roland also mentioned in the comments, the GIL makes e.g. debugging of the garbage collector much easier. There are implementations like IronPython which don't have a GIL though.

Python 2 and Python 3 have different approaches how a thread acquires the GIL.

Python 2: http://www.dabeaz.com/python/GIL.pdf

Python 3: http://www.dabeaz.com/python/NewGIL.pdf

One solution is to spawn independent Python interpreters. They work as separate processes and can do their own work. The multiprocessing module can help you here.

edited Sep 6, 2015 at 17:33

answered Sep 6, 2015 at 2:03

HelloWorld

2,0144 gold badges40 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Andrea Corbellini Over a year ago

"One solution to avoid"? Probably you meant "one possible solution".

A. Jesse Jiryu Davis Over a year ago

I edited the answer, I believe @AndreaCorbellini is correct.

Roland Smith Over a year ago

Those historical reasons were and still are extremely practical and valid. Having the GIL makes the garbage collection code a lot simpler (it is not thread-safe, but would have to be so without the GIL). The desire to remove the GIL has come up several times, but no one has managed it, for several reasons.

HelloWorld Over a year ago

AndreaCorbellin is right, yes, but next time give the OP please some time. I prefer to correct my answer on my own. The fastest gun in the west does also apply on editing. @RolandSmith: That's true. I didn't want to imply the GIL is just bad. I added this to my answer.

Collectives™ on Stack Overflow

python multiple thread performance

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related