instantiate python object within a c function called via ctypes

Question

My embedded Python 3.3 program segfaults when I instantiate python objects from a c function called by ctypes.

After setting up the interpreter, I can successfully instantiate a python Int (as well as a custom c extension type) from c main:

#import <Python/Python.h>  

#define LOGPY(x) \
{ fprintf(stderr, "%s: ", #x); PyObject_Print((PyObject*)(x), stderr, 0); fputc('\n', stderr); }

// c function to be called from python script via ctypes.
void instantiate() {
  PyObject* instance = PyObject_CallObject((PyObject*)&PyLong_Type, NULL);
  LOGPY(instance);
}

int main(int argc, char* argv[]) {
  Py_Initialize();
  instantiate(); // works fine

  // run a script that calls instantiate() via ctypes.
  FILE* scriptFile = fopen("emb.py", "r");
  if (!scriptFile) {
    fprintf(stderr, "ERROR: cannot open script file\n");
    return 1;
  }

  PyRun_SimpleFileEx(scriptFile, scriptPath, 1); // close on completion
  return 0;
}

I then run a python script using PyRun_SimpleFileEx. It appears to run just fine, but when it calls instantiate() via ctypes, the program segfaults inside PyObject_CallObject:

import ctypes as ct
dy = ct.CDLL('./emb')
dy.instantiate() # segfaults

lldb output:

instance: 0
Process 52068 stopped
* thread #1: tid = 0x1c03, 0x000000010000d3f5 Python`PyObject_Call + 69, stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x000000010000d3f5 Python`PyObject_Call + 69
Python`PyObject_Call + 69:
-> 0x10000d3f5:  movl   24(%rax), %edx
   0x10000d3f8:  incl   %edx
   0x10000d3fa:  movl   %edx, 24(%rax)
   0x10000d3fd:  leaq   2069148(%rip), %rax       ; _Py_CheckRecursionLimit
(lldb) bt
* thread #1: tid = 0x1c03, 0x000000010000d3f5 Python`PyObject_Call + 69, stop reason = EXC_BAD_ACCESS (code=1, address=0x18)
    frame #0: 0x000000010000d3f5 Python`PyObject_Call + 69
    frame #1: 0x00000001000d5197 Python`PyEval_CallObjectWithKeywords + 87
    frame #2: 0x0000000201100d8e emb`instantiate + 30 at emb.c:9

Why does the call to instantiate() fail from ctypes only? The function only crashes when it calls into the python lib, so perhaps some interpreter state is getting munged by the ctypes FFI call?

How does dy_test() get to instantiate? Your log message, if it comes via instantiate, shows that PyObject_CallObject went to completion. Otherwise you wouldn't be seeing the LOGPY output. Can you try a PyLongObject or some other standard callable in Python? That would tell you if it was a problem with your type setup. — Andrew Dalke
– Andrew Dalke, Commented Dec 4, 2012 at 19:42
sorry, i renamed dy_test for clarity in the question; fixed. passing &PyLong_Type fails in the same way; i'll reduce the repro accordingly. — gwk
– gwk, Commented Dec 4, 2012 at 22:41
Yes! PyDLL works - docs say the only difference is that the GIL remains locked and it checks for errors at the end. Want to write up an answer or should I? — gwk
– gwk, Commented Dec 4, 2012 at 23:07

gwk · Accepted Answer · 2012-12-28 16:40:01Z

3

Thanks to Armin Rigo for the hint. The problem is that libraries loaded via ctypes.CDLL() create functions that release the GIL when calling into the native code. As far as I can tell this means that in order for the native function to call back into python code, it would need to acquire the GIL first using the python C API.

The easier alternative is to use ctypes.PyDLL(), which does not release the GIL (it also checks the python error flag). The documentation says, "Thus, this is only useful to call Python C API functions directly." My code is more indirect, in that I have python code calling into my own C functions, which then call into the python C API, but the problem is the same.

answered Dec 28, 2012 at 16:40

gwk

7267 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

instantiate python object within a c function called via ctypes

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related