How to create and use a python object stored as a c++ object?

Question

So as the title sort of suggests, I am working on a c++ project where I need to call a python module, save it as an object, and call one of its methods multiple times. Below you can find the code for the class, which contains the python object. Currently, it is implemented inside of a for loop which calls the method multiple times. Instantiating the class works fine, along with the first call to the class. However, upon completion of the first loop of the for loop the program crashes with an error along the lines of "free(): invalid size" or sometimes "double free or corruption". I tried using valgrind to try and track down the memory leak, but I get a lot of traces to pythonCApi calls that I dont really understand.

#include <python2.7/Python.h>
#include <iostream>
#include <algorithm>
#include "predictor.hpp"

using namespace std;


predictor::predictor()
{
  Py_Initialize();

  pName = PyString_FromString("predictor");

  pModule = PyImport_Import(pName);

  Py_XDECREF(pName);

  if (pModule == nullptr) {
    PyErr_Print();
    std::cerr << "Fails to import the module predictor, check installation.\n";
  }

  // dict is a borrowed reference.
  dict = PyModule_GetDict(pModule);
  if (dict == nullptr) {
    PyErr_Print();
    std::cerr << "Fails to get the dictionary, check predictor installation.\n";
    Py_XDECREF(pModule);
  }
  Py_XDECREF(pModule);

  // Builds the name of a callable class
  python_class = PyDict_GetItemString(dict, "Predictor");
  if (python_class == nullptr || python_class == NULL) {
    PyErr_Print();
    std::cerr << "Fails to get the Python class, check predictor installation.\n";
    Py_XDECREF(dict);
  }
  Py_XDECREF(dict);

  // Creates an instance of the class
  if (PyCallable_Check(python_class)) {
    object = PyObject_CallObject(python_class, nullptr);
    if (object == NULL)
    {
        cerr << "Fails to create object.";
        Py_XDECREF(python_class);
    }
    Py_XDECREF(python_class);
} else {
    PyErr_Print();
    std::cout << "Cannot instantiate the Python class" << std::endl;
    Py_XDECREF(python_class);
}

pMethod = PyString_FromString("predict_all");

}


predictor::~predictor()
{

  Py_XDECREF(pMethod);

  Py_XDECREF(object);
  Py_Finalize();

}


long predictor::predict(string rule)
{

  PyObject *pRule = PyString_FromString(rule.c_str());
  PyObject *value = PyObject_CallMethodObjArgs(object, pMethod, pRule, NULL);

  long endValue = PyInt_AsLong(value);

  if (endValue == -1)
  {
    if(!PyErr_Occurred())
    {

        PyErr_Print();
        cerr << "";

        Py_XDECREF(value);
        Py_XDECREF(pRule); 

        return NULL;
    }

    //PyErr_Print();

  }

  Py_XDECREF(value);
  Py_XDECREF(pRule); 

  return endValue;}

horstr · Accepted Answer · 2020-05-13 09:31:15Z

2

The most critical part of writing Python C/C++ code is getting the reference counting right. Python differentiates between different kinds of references, namely new, stolen and borrowed references.

With every API function you call, you have to the check the documentation to see what kind of reference it returns, if any.

New references belong to the caller, so it's correct to use Py_XDECREF to free the object by decrementing the reference count. Be sure not to call Py_XDECREF more than once unless you incremented the reference count in between. In your error handling, Py_XDECREF(pModule) happens twice, for example, because you do not return in the error case, you simply continue.

Borrowed references are owned by somebody else and the reference count was not incremented for you. So, calling Py_XDECREF is only valid if you incremented the reference count yourself before doing so.

PyModule_GetDict(pModule) returns a borrowed reference. You do not increment the reference count, yet you decrement it later with Py_XDECREF(dict). The same is true for PyDict_GetItemString(dict, "predictor"), it returns a borrowed reference yet you decrement it with Py_XDECREF(python_class).

My assumption is that in both cases (dict, python_class), the borrowed references are owned by the module pModule you imported with PyImport_Import(pName). So it's highly likely that you must not decrement the pModule reference count for as long as you're working with borrowed references owned by pModule. Release pModule with Py_XDECREF once you're not using those borrowed references anymore. Alternatively, you could increment the reference count of the borrowed references but as long as you keep pModule around, that should not be necessary.

edited May 13, 2020 at 9:31

answered May 12, 2020 at 7:42

horstr

2,87718 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Daniel Bresnahan Over a year ago

Okay, so your advice solved the first memory error (by removing all the DECREFS to borrowed code, and removing the repeated decrefs). However, after that I get an error that says free() invalid size. Is that related to my code here, or something else.

horstr Over a year ago

I recreated your sample. When I remove the incorrect PyXDECREFs for dict and python_class, I cannot reproduce memory issues anymore. valgrind does report memor errors. Before, it did. I also moved the pModule PyXDECREF after the last use of a borrowed reference logically owned by it - so after the instance creation block. There were no identifiable errors in that case, but based on the ownership, this is the correct thing to do. Many samples seem to keep the pModule reference around until deinitialization, but in terms of references, I don't see a need for that.

Collectives™ on Stack Overflow

How to create and use a python object stored as a c++ object?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related