Python Ctypes register callback functions

Question

I ran into something very strange using Python and ctypes. I'm using Python 3.4.3. First, some background into the project:

I have compiled a custom dll from C code. I'm using ctypes to interface with the dll. The C library is interfacing with some custom hardware. Sometimes the hardware generates an interrupt and passes it along to the C library on the computer. In the C API, there is a function with the prototype void register_callback(int addr, void (*callback)(void)). I have an array of callback function pointers, which are initialized to NULL. When this function is called, the callback function pointer at index addr is set to callback, like this: callbacks[addr] = callback;.

When the user programs in Python, they instantiate objects from classes that model different hardware parts (such as a button or an RGB LED). They can then write a custom callback function and call button.register_callback(func) (assuming they have a Button object named button, of course), which calls the register_callback function in the C library. Now, when the button is pressed and the interrupt is generated, the C library will call the appropriate callback function (i.e. callbacks[addr]();).

Now, the weirdness:

In Python, my first attempt at the register_callback method in Python looked like this:

class Obj:
    def __init__(self, name):
        # Initialize stuff

    def register_callback(self, func):
        CB_T = ctypes.CFUNCTYPE(None)
        cb_ptr = CB_T(func)
        host_api.register_callback(self.addr, cb_ptr) # host_api is the loaded dll

And in main:

def cb1():
    print("cb1")

def cb2():
    print("cb2")

def main(argv):
    # Initialization stuff
    # Now create the objects and register the callbacks:
    obj = Obj_module.Obj()
    obj2 = Obj_module.Obj()
    obj.register_callback(cb1)
    obj2.register_callback(cb2)

    while True:
        pass

When I ran this, only "cb2" was being printed, regardless of which button I was pressing. The REALLY weird thing is that when I switched the order in which I registered the callbacks :

    obj2.register_callback(cb2)
    obj.register_callback(cb1)

only "cb1" was being printed, regardless of the button I pressed! In the C library, I verified (by printf) that different callback function pointers were being set and called, depending on the button, but that the same function pointer was being passed to the C register_callback function.

I was able to fix the problem by adding a line to the register_callback method:

def register_callback(self, func):
    CB_T = ctypes.CFUNCTYPE(None)
    cb_ptr = CB_T(func)
    (ctypes.cast(cb_ptr, ctypes.POINTER(ctypes.c_int)))
    host_api.register_callback(self.addr, cb_ptr)

Apparently, converting cb_ptr to a ctypes POINTER fixed the problem - different function pointers were being passed in, and I successfully saw "cb1" or "cb2" printed, depending on the button that I pressed.

My question is, WHY? Why was the same function pointer being passed in the original code, why was it changing depending on the order that I registered the callbacks, and why does converting cb_ptr to a ctypes POINTER ensure that the function pointers are different?

I'm kind of a beginner at Python, but I'm much more experienced at C. Thanks in advance for your responses.

Alden · Accepted Answer · 2016-07-08 23:13:01Z

5

Your cb_ptr is being garbage collected. From the documentation:

Make sure you keep references to CFUNCTYPE() objects as long as they are used from C code. ctypes doesn’t, and if you don’t, they may be garbage collected, crashing your program when a callback is made.

In this code sample if the line ptrs.append(cb_ptr) is commented out the location of cb_ptr is the same for both Obj instances (on my computer). Uncommenting the line results in two memory locations.

import ctypes

ptrs = []

class Obj:
    def __init__(self):
        pass

    def register_callback(self, func):
        CB_T = ctypes.CFUNCTYPE(None)
        cb_ptr = CB_T(func)
        ptrs.append(cb_ptr)
        print(cb_ptr)

def cb1(): print("cb1")

def cb2(): print("cb2")

def main(argv):
    obj = Obj()
    obj2 = Obj()
    obj.register_callback(cb1)
    obj2.register_callback(cb2)

main(None)

answered Jul 8, 2016 at 23:13

Alden

2,2691 gold badge15 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Paul Cornelius Over a year ago

Great insight and well explained.

mgarey Over a year ago

Great example. I just want to make sure I understand: I'm assuming that ctypes.CFUNCTYPE actually returns an object, but when that object goes out of scope it is garbage collected. But when I cast it to a pointer, it actually passed the address to the C function, so although cb_ptr (which is now out of scope) is garbage collected, C still has the correct address. Why doesn't the last instance of cb_ptr get garbage collected? Isn't it out of scope, too?

Alden Over a year ago

What you are experiencing is undefined behavior, so I can only speculate what is going on. When the CFUNCTYPE object is created a callback function that is callable from C code is created which will then call cb1 or cb2. When the object is garbage collected the data for this callback function still persists in memory. In your original case, I suspect that the second callback function overwrote the same location as the first one, so both C functions ended up executing the same code. When cast() is used for one reason or another the second call does not overwrite the first.

Alden Over a year ago

So in both cases the C code contains a pointer to an invalid address, but in the one where cast() is used the python interpreter just happened to not overwrite the previous callback function.

Collectives™ on Stack Overflow

Python Ctypes register callback functions

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related