Python C API crashes on 'import numpy' when initilizing multiple times

Question

While working with the Python C API, I found that the python interpreter crashes when initializing it a second time and executing import numpy after each initilization. Any other command (e.g. import time) will do just fine.

#include <Python.h>

int main(int argc, char ** argv)
{
    while(1){
        printf("Initializing python interpreter...\n");
        Py_Initialize();
        if(PyRun_SimpleString("import numpy")) {
            exit(1);
        }
        printf("Finalizing python interpreter...\n");
        Py_Finalize();
    }
    return 0;
}

The above program crashes on both of my test systems (ubuntu and manjaro, no matter what python version I use) with a Segmentation Fault while executing import numpy a second time.

The Documentation (https://docs.python.org/3/c-api/init.html?#c.Py_FinalizeEx) indeed says that: Some extensions may not work properly if their initialization routine is called more than once; this can happen if an application calls Py_Initialize() and Py_FinalizeEx() more than once.

But shouldn't there be a way to properly clear the memory of the interpreter so it can be initialized multiple times? For example if I have a program that allows the user to run a custom python script, it should be possible to run the same script multiple times without restarting the program. Any clues?

It looks like this has been pretty widely reported before stackoverflow.com/questions/7676314/… stackoverflow.com/questions/14843408/… stackoverflow.com/questions/16779799/… — DavidW
– DavidW, Commented Dec 13, 2019 at 8:22

ShadowRanger · Accepted Answer · 2019-12-13 00:52:13Z

You can't just declare that all memory should be cleared/reset; Python isn't in control of all of it. Extension modules are actual shared object/DLL files, that can do literally anything a plain C program can do, and they're not required to register all of their actions in such a way that the core Python interpreter knows how to undo them on finalization. Python can't know that this part of the SO/DLL memory stores data that must be cleared, while that part is static data that should be left alone.

It's perfectly possible to run a script multiple times, but you don't do it by finalizing and reinitializing, you just actually run the script multiple times in a single initialization (and hope it's written in an idempotent fashion).

As an alternative, if you're on a UNIX-like box (read: anything but Windows), "resetting" can be done a different way, by execing your program to restamp it with a fresh run. This resets more thoroughly than Python itself can, though even then it's not foolproof; e.g. if file descriptors aren't opened in O_CLOEXEC mode, they'll stay open in the newly exec-ed process.

Collectives™ on Stack Overflow

Python C API crashes on 'import numpy' when initilizing multiple times

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related