Creating a numpy array in C from an allocated array is causing memory leaks

Question

I have traced a memory leak in my program to a Python module I wrote in C to efficiently parse an array expressed in ASCII-hex. (e.g. "FF 39 00 FC ...")

char* buf;
unsigned short bytesPerTable;
if (!PyArg_ParseTuple(args, "sH", &buf, &bytesPerTable))
{
    return NULL;
}

unsigned short rowSize = bytesPerTable;
char* CArray = malloc(rowSize * sizeof(char));

// Populate CArray with data parsed from buf
ascii_buf_to_table(buf, bytesPerTable, rowSize, CArray);

int dims[1] = {rowSize};

PyObject* pythonArray = PyArray_SimpleNewFromData(1, (npy_intp*)dims, NPY_INT8, (void*)CArray);
return Py_BuildValue("(O)", pythonArray);

I realized that numpy does not know to free the memory allocated for CArray, thus causing a memory leak. After some research into this issue, at the suggestion of comments in this article I added the following line which is supposed to tell the array that it "owns" its data, and to free it when it is deleted.

PyArray_ENABLEFLAGS((PyArrayObject*)pythonArray, NPY_ARRAY_OWNDATA);

But I am still getting the memory leak. What am I doing wrong? How do I get the NPY_ARRAY_OWNDATA flag to work properly?

For reference, the documentation in ndarraytypes.h makes it seem like this should work:

/*
 * If set, the array owns the data: it will be free'd when the array
 * is deleted.
 *
 * This flag may be tested for in PyArray_FLAGS(arr).
 */
#define NPY_ARRAY_OWNDATA         0x0004

Also for reference, the following code (calling the Python function defined in C) demonstrates the memory leak.

tableData = "FF 39 00 FC FD 37 FF FF F9 38 FE FF F1 39 FE FC \n" \
            "EF 38 FF FE 47 40 00 FB 3D 3B 00 FE 41 3D 00 FE \n" \
            "43 3E 00 FF 42 3C FE 02 3C 40 FD 02 31 40 FE FF \n" \
            "2E 3E FF FE 24 3D FF FE 15 3E 00 FC 0D 3C 01 FA \n" \
            "02 3E 01 FE 01 3E 00 FF F7 3F FF FB F4 3F FF FB \n" \
            "F1 3D FE 00 F4 3D FE 00 F9 3E FE FC FE 3E FD FE \n" \
            "F6 3E FE 02 03 3E 00 FE 04 3E 00 FC 0B 3D 00 FD \n" \
            "09 3A 00 01 03 3D 00 FD FB 3B FE FB FD 3E FD FF \n"

for i in xrange(1000000):
    PES = ParseTable(tableData, 128, 4) //Causes memory usage to skyrocket

Ulfalizer · Accepted Answer · 2015-03-06 20:10:22Z

8

It's probably a reference-count issue (from How to extend NumPy):

One common source of reference-count errors is the Py_BuildValue function. Pay careful attention to the difference between the ‘N’ format character and the ‘O’ format character. If you create a new object in your subroutine (such as an output array), and you are passing it back in a tuple of return values, then you should most- likely use the ‘N’ format character in Py_BuildValue. The ‘O’ character will increase the reference count by one. This will leave the caller with two reference counts for a brand-new array. When the variable is deleted and the reference count decremented by one, there will still be that extra reference count, and the array will never be deallocated. You will have a reference-counting induced memory leak. Using the ‘N’ character will avoid this situation as it will return to the caller an object (inside the tuple) with a single reference count.

edited Mar 6, 2015 at 20:10

answered Mar 6, 2015 at 19:52

Ulfalizer

4,7921 gold badge23 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

dpitch40 Over a year ago

Is there a way to check the reference count of Python objects to confirm this?

Ulfalizer Over a year ago

@dpitch40: Not sure off the top of my head. Some quick googling didn't reveal anything.

Ulfalizer Over a year ago

Found this re. getting the reference count by the way: stackoverflow.com/questions/26134455/…. Been a long while since I did any Python/C interfacing. :)

Ulfalizer Over a year ago

(I edited the answer to just the How to extend NumPy text after I found it by the way. Felt a bit bad making it look like the comment was re. that. :P)

Jaime Over a year ago

You can check on the Python side with sys.getrefcount. If that is indeed the source of the leak, you can simply return pythonArray, which is already a PyObject*, no need to Py_BuildValue when returning a single item.

|

Collectives™ on Stack Overflow

Creating a numpy array in C from an allocated array is causing memory leaks

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related