Embedding multiple Python sub-interpreters into a C program

Question

I am writing a C program that spawns multiple C threads, with one Python sub-interpreter per thread. The sub-interpreters do not share any mutable Python variables, they are isolated from each other. (They do have a read-only access to a common PyObject (immutable) that is exposed from the main() function in the C program).

Is this possible in Python 3.7 or 3.8, without sharing GIL between the sub-interpreters?

Here is the pseudo-code of what I have been trying:

void *spawnInterpreter(void* p) {
    …
    PyThreadState* save_tstate = PyThreadState_Swap(NULL);
    PyThreadState* tstate = Py_NewInterpreter();
    PyThreadState_Swap(save_tstate);

    //do some Python work (with variables that are NOT shared with other thread’s sub-interpreter
    PyRun_SimpleString( . . .);
    . . . 
}


int main() {
...
    pthread_create(&thread1, NULL, spawnInterpreter,  “in1”);
    pthread_create(&thread2, NULL, spawnInterpreter, "in2");
...
}

I could get this to work in 3.6 (without acquiring GIL or managing PyThreadState in C threads), but in Python 3.7 I get:

[New Thread 0x7ffff5f78700 (LWP 16392)]
Fatal Python error: drop_gil: GIL is not locked

The "Fatal Python error" you got looks like a regression. Some related things changed in the Python 3.7 runtime code, but I would not expect them to have changed any behavior. I strongly recommend opening an issue on bugs.python.org (and post the link here). — Eric Snow
– Eric Snow, Commented Apr 26, 2019 at 17:42
I'll provide an answer regarding subinterpreters not sharing the GIL. — Eric Snow
– Eric Snow, Commented Apr 26, 2019 at 17:43
PyThreadState_Swap() must be called with the GIL held and does not release it. Likewise with Py_NewInterpreter(). So at the point you call PyRun_SimpleString(), you are holding the GIL. Everything should be fine. If I had to guess, I'd suggest it's related to the GILState API, which doesn't play nice with subinterpreter yet (see bugs.python.org/issue10915 and bugs.python.org/issue15751). — Eric Snow
– Eric Snow, Commented Apr 26, 2019 at 17:50
Do you have the stack trace for the fatal error? Where is drop_gil() being called without the GIL being held? It would be in Python/ceval.c, but which call specifically? — Eric Snow
– Eric Snow, Commented Apr 26, 2019 at 17:50

Eric Snow · Accepted Answer · 2019-04-26 18:42:25Z

21

Unfortunately, subinterpreters still share the GIL in 3.7 and 3.8. This is something I'm personally working on changing. See PEP 554 and my multi-core Python project. I'm also giving a talk at PyCon next week that covers the topic in some detail.

My hope has been to make it possible in Python 3.8, but it's looking more likely for 3.9 at this point. The main challenge is that the C-API and CPython runtime are not thread-safe. While most of the C-API and runtime can switch to using the per-interpreter GIL, other things will have to change in that scenario:

some process-global resources have to be managed more carefully without a GIL (e.g. env vars, file handles)
there is global runtime state that interpreters must continue to share, so that much must still be protected by a global lock (though one which will not need to block the Python bytecode eval loop)
some of the global runtime state needs to be moved down to per-interpreter state (e.g. GC, memory allocators, warnings)
objects will need to be strictly per-interpreter (for now), so the C-API must be strict in not allowing objects to cross the interpreter boundary
parts of the C-API that are not specific to an interpreter context must change to no longer require holding the GIL

The problem is tractable, but it takes time to apply the necessary care when working on such critical code. Hence the likely target of 3.9.

Regardless, I'm thankful that you've posted here. Most of my efforts have focused on the impact on Python code, rather than the C-API (e.g. embedders). So feedback on how my project relates to use of subinterpreters via the C-API is super helpful. For instance, one thing you've reminded me of is that creating subinterpreters via the C-API is slightly different than the equivalent in PEP 554. That needs to be considered more carefully. Also, PEP 554 exposes virtually none of its additions in the C-API. That's probably okay, but interacting with channels from the C-API might be valuable in the short term.

answered Apr 26, 2019 at 18:42

Eric Snow

1,2189 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Marti Nito Over a year ago

I think it will then be a PIL (or PIGIL?!?), not a GIL ¯\_(ツ)_/¯. @eric: Nice to hear that there is some work on the GIL front :)

Eric Over a year ago

Eric, your PEP 554 indicates version 3.10 but there's nothing in 3.10.0rc1 release notes. What's the target?

Collectives™ on Stack Overflow

Embedding multiple Python sub-interpreters into a C program

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related