Strange memory behaviour when using Python C API

Question

I am trying to implement a Python wrapper using the Python C API over a C++ library. I need to implement conversions so I can use objects in Python and C++. I already done that in the past but I have an error I really have a hard time with.

I have a very basic test function:

PyObject* convert_to_python() {
    std::cout << "Convert to PyObject" << std::endl;
    long int a = 20;
    PyObject* py_a = PyInt_FromLong(a);
    std::cout << "Convert to PyObject ok" << std::endl;
    return py_a;
}

I call this function inside a GoogleTest macro:

TEST(Wrapper, ConvertTest) {
    PyObject *py_m = convert_to_python();
}

And my output is:

Convert to PyObject
Segmentation fault (core dumped)

I also ran valgrind on it:

valgrind --tool=memcheck --track-origins=yes --leak-check=full ./my_convert

But it doesn't give me much information about it:

Invalid read of size 8
==19030==    at 0x4F70A7B: PyInt_FromLong (in /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0)
==19030==    by 0x541E6BF: _object* pysmud_from<float>(smu::Matrix<float, 0, 0>&) (smu_type_conversions.cpp:308)
==19030==    by 0x43A144: (anonymous namespace)::Wrapper_ConvertMatrix_Test::Body() (test_wrapper.cpp:12)
==19030==    by 0x43A0C6: (anonymous namespace)::Wrapper_ConvertMatrix_Test::TestBody() (test_wrapper.cpp:10)
==19030==    by 0x465B4D: void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2078)
==19030==    by 0x460684: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
==19030==    by 0x444C05: testing::Test::Run() (gtest.cc:2151)
==19030==    by 0x4454C9: testing::TestInfo::Run() (gtest.cc:2326)
==19030==    by 0x445BEA: testing::TestCase::Run() (gtest.cc:2444)
==19030==    by 0x44CF41: testing::internal::UnitTestImpl::RunAllTests() (gtest.cc:4315)
==19030==    by 0x46712C: bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2078)
==19030==    by 0x461532: bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (gtest.cc:2114)
==19030==  Address 0x0 is not stack'd, malloc'd or (recently) free'd

I think this code should work but I can't get what it's wrong with what I wrote. Did I wrongly included or linked Python files and libraries ?

EDIT: Gives no errors

#include <Python.h>

PyObject* convert_long_int(long int a) {
  PyObject *ret = PyInt_FromLong(a);
  return ret;
}

int main(void) {
  long int a = 65454984;
  PyObject *pya = convert_long_int(a);
  return 0;
}

If compiling with gcc -o wraptest -I/usr/include/python2.7 wraptest.c -L/usr/lib/x86_64-linux-gnu/ -lpython2.7

What does the initialization do ?

You know that a cout statement after a return will never be shown right ? Is it a typo :p ? — Xatyrian
– Xatyrian, Commented Feb 22, 2018 at 9:35
It's a mistake from me the return is before. I misplaced it when writing the question my bad. — baptiste
– baptiste, Commented Feb 22, 2018 at 9:38

Olaf Dietsche · Accepted Answer · 2018-02-22 10:37:28Z

I can confirm the segmentation fault on Ubuntu 16.04 and Python 2.7, if I omit the initialization.

Looking at Embedding Python in Another Application, there's this example

#include <Python.h>

int
main(int argc, char *argv[])
{
  Py_SetProgramName(argv[0]);  /* optional but recommended */
  Py_Initialize();
  PyRun_SimpleString("from time import time,ctime\n"
                     "print 'Today is',ctime(time())\n");
  Py_Finalize();
  return 0;
}

So when I do an equivalent minimal main

int main()
{
    Py_Initialize();
    PyObject *p = convert_to_python();
    Py_Finalize();
    return 0;
}

it works without crash.

The difference between the two examples is

long int a = 20;

and

long int a = 65454984;

I guess, it has to do with PyInt_FromLong(long ival)

The current implementation keeps an array of integer objects for all integers between -5 and 256, when you create an int in that range you actually just get back a reference to the existing object.

Maybe Python tries to access an uninitialized pointer or memory range without the initialization.

When I change the example using a = 256, it crashes. Using a = 257, it doesn't.

Looking at cpython/Objects/intobject.c:79, you can see an array of pointers

static PyIntObject *small_ints[NSMALLNEGINTS + NSMALLPOSINTS];

which is accessed right below in PyInt_FromLong(long ival)

v = small_ints[ival + NSMALLNEGINTS];
Py_INCREF(v);

But without initialization from _PyInt_Init(void)

for (ival = -NSMALLNEGINTS; ival < NSMALLPOSINTS; ival++) {
    if (!free_list && (free_list = fill_free_list()) == NULL)
        return 0;
    /* PyObject_New is inlined */
    v = free_list;
    free_list = (PyIntObject *)Py_TYPE(v);
    (void)PyObject_INIT(v, &PyInt_Type);
    v->ob_ival = ival;
    small_ints[ival + NSMALLNEGINTS] = v;
}

these pointers are all NULL, causing the crash.

I tried a simple main and compiling with gcc -o wraptest -I/usr/include/python2.7 wraptest.c -L/usr/lib/x86_64-linux-gnu/ -lpython2.7 gives no error as well. But in this case I don't have initialization as well.

Collectives™ on Stack Overflow

Strange memory behaviour when using Python C API

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related