Why do Python functions get garbage collected?

Question

I have a C++ library which uses Python callbacks. The callback, i.e. PyObject*, is stored in an object of class UnaryFunction, and the constructor Py_INCREFs it. The destuctor Py_XDECREFs it. That's the problem. The interpreter segfaults on that DECREF.

My solution is to just not DECREF it, but that seems wrong. What is the proper way to INC/DEC the reference count of a function, and more importantly, why does the interpreter try to GC a function body when there are other live references to it?

Edit: On Linux instead of a segfault I get an assertion fail that says:

python: Objects/funcobject.c:442: func_dealloc: Assertion 'g->gc.gc_refs != (-2)' failed.

This seems like a question for the mailing list. Google "Python mailing list". On there you can find people who know Python very well, sometimes even the creators/devs of Python themselves. — Sunjay Varma
– Sunjay Varma, Commented Sep 8, 2011 at 2:06
@J.F.Sebastian, good guess but the method does not go out of scope. The problem is exhibited in a simple flat script with a simple `def myfunc(x):". If I comment out the parts that use the callback then I can call myfunc() through the end of the script. — Adam
– Adam, Commented Sep 8, 2011 at 20:45

brandizzi · Accepted Answer · 2011-09-08 11:13:54Z

1

A crash does not necessary mean that it is trying to GC an used object. It can also mean that you are calling python code without the interpretor lock.

Calling Py_XDECREF in a destructor leads me to think you have something like this:

void MyCallback(myfunc, myarg)
{
    ...
    PyGILState_STATE gilstate = PyGILState_Ensure();
    try {
            myfunc(myarg);
    } catch (...) {
        ...
    }
    PyGILState_Release(gilstate);

    // myfunc goes out of scope here --> CRASH because we no longer own the GIL
}

with the simple solution:

...
try {
    scopefunc = myfunc;
    myfunc = emptyfunc();
    scopefunc(myarg);
} ...

edited Sep 8, 2011 at 11:13

brandizzi

27.4k9 gold badges111 silver badges171 bronze badges

answered Sep 8, 2011 at 3:47

Mihai Stan

1,0626 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Adam Over a year ago

Actually, I do not touch the GIL at all. From what I understand the GIL does not get released unless it's explicitly released. My C++ code simply uses PyEval_CallObject() to call the function. I say it's collected because the segfault is: Objects/funcobject.c:442: func_dealloc: Assertion 'g->gc.gc_refs != (-2)' failed. and that's talking about a dealloc and GC references.

Mihai Stan Over a year ago

If you have just one thread then things should be simpler. You have to find the mismatched incref/decref, and calling sys.getrefcount before and after each step that involves the function could give you some clues

Adam · Accepted Answer · 2011-09-09 00:12:35Z

0

It appears the Py_INCREF simply doesn't actually increment the refcount.

answered Sep 9, 2011 at 0:12

Adam

17.4k8 gold badges55 silver badges103 bronze badges

Collectives™ on Stack Overflow

Why do Python functions get garbage collected?

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related