cython wrap cpp structs and function with parameter an array of structs

Question

I am using Cython to wrap a c++ library in python. Unfortunately, I have no access to the c++ library. Therefore, I have to find somehow a way to wrap the structures and functions as exposed by the lib API.

My question regards the best way to wrap a c++ structure and; subsequently, how to create a memory view in python and pass its pointer (first element address) to a c++ function with parameter an array of cpp structures.

For example, let say I have the following h file:

//test.h

struct cxxTestData
{
   int m_id;
   double m_value;
};

void processData(cxxTestData* array_of_test_data, int isizeArr)

My pyx file will look like the following

cdef extern from "Test.h":
    cdef struct cxxTestData:
        int m_id
        double m_value

cdef class pyTestData:
    cdef cxxTestData cstr

    def __init__(self, id, value):
        self.cstr.m_id = id
        self.cstr.m_value = value

    @property
    def ID(self):
        return self.cstr.m_id

    @property
    def Value(self):
        return self.cstr.m_value

Now, I want to create a number of pyTestData and store them in an array of dtype object. Then I want to pass this array as memory view in a cython/python function.

The wrapping function will have the following signature

cpdef void pyProcessData(pyTestData[::1] test_data_arr)

I have tested the above and it compiles successfully. I managed also to modify the members of each structure. However, this is not what I am trying to achieve. My question is how from this point I can pass an array with the c++ structures encapsulated in each pyTestData object (via the self.cstr).

As an example please have a look to the following listing:

cpdef void pyProcessData(pyTestData[::1] test_data_arr):
    cdef int isize test_data_arr.shape[0]

    # here I want to create/insert an array of type cxxTestData to pass it 
    # to the cpp function
    # In other words, I want to create an array of [test_data_arr.cstr]
    # I guess I can use cxxTestData[::1] or cxxTestData* via malloc and
    # copy each test_data_arr[i].cstr to this new array
    cdef cxxTestData* testarray = <cxxTestData*>malloc(isize*sizeof(cxxTestData))

    cdef int i
    for i in range(isize):
        testarray[i] = test_data_arr[i].cstr
    processData(&testarray[0], isize)

    for i in range(isize):
        arrcntrs[i].pystr = testarray[i]

    free(testarray)

Has anyone come across with such a case? Is there any better way to pass my python objects in the above function without having to copy over the cxx structures internally?

Many thanks in advance and apologies if I do something fundamentally wrong.

1) I don't think pyTestData[::1] really works - it will actually accept any array of objects without complaint. I'm sure it didn't used to compile at all, so compiling and not doing what you want feels like a regression. 2) The fundamental issue is that pyTestData are Python objects, so must be allocated individually and also contain Python reference counting data. Therefore, there isn't a solid block of c++ objects for you to send to your function (without copying). — DavidW
– DavidW, Commented Feb 12, 2019 at 22:01
Many thanks for your answer. Actually, pyTestData[::1] does compile. However, I agree that it accepts any array of objects. Therefore, not the best for storing my objects. Do I have to define the dtype explicitly (pyTestDataType = [('m_id', 'int'), ('m_value', 'double')]) and then use it to my signature so as to accept only these objects? Could you provide an example? This is not the real question; however, it could help me to do all things correct. — GioR
– GioR, Commented Feb 13, 2019 at 6:47
Unfortunately I don't really have a good solution to this... — DavidW
– DavidW, Commented Feb 13, 2019 at 6:54

DavidW · Accepted Answer · 2019-02-19 21:01:41Z

1

Since you want an array of cxxTestData to pass to your C++ functions, the best thing to do is to allocate it as an array. Some untested code that illustrates the approach:

cdef class TestDataArray:
    cdef cxxTestData* array:
    def __init__(self, int length):
        self.array = <cxxTestData*>calloc(length,sizeof(cxxTestData))
    def __dealloc__(self):
        free(self.array)
    def __getitem__(self, int idx):
        return PyTestData.from_pointer(&self.array[idx],self) # see later
    def __setitem__(self, int idx, PyTestData pyobj): # this is optional
        self.array[idx] = deref(pyobj.cstr)

You then want to slightly modify your PyTestData class so that it holds a pointer rather than holding the class directly. It should also have a field representing the ultimate owner of the data (e.g. the array). This ensures the array is kept alive, and can also allow for the case where the PyTestData owns its own data:

cdef class PyTestData:
    cdef cxxTestData* cstr
    cdef object owner

    def __init__(self, id, value):
        self.owner = None
        self.cstr = <cxxTestData*>malloc(sizeof(cxxTestData))
        self.cstr.m_id = id
        self.cstr.m_value = value

    def __dealloc__(self):
        if self.owner is None: # i.e. this class owns it
             free(self.cstr)

    @staticmethod
    cdef PyTestData from_pointer(cxxTestData* ptr, owner):
        # calling __new__ avoids calling the constructor
        cdef PyTestData x = PyTestData.__new__(PyTestData)
        x.owner = owner
        x.cstr = ptr
        return x

There is a little extra effort in creating the TestDataArray class, but it stores the data in a format directly usable from C++, and so I think it's the best solution.

edited Feb 19, 2019 at 21:01

answered Feb 14, 2019 at 18:34

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

GioR Over a year ago

Hi David, many thanks for the illustrative example. So, the TestDataArray is an additional wrapper class that stores my cxx structures, right? And with a setitem I can fill my array.

DavidW Over a year ago

Yes, TestDataArray is an additional wrapper. I'd probably use __getitem__ and then change the data in place instead of using __setitem__ (just because __setitem__ involves more copying, which you seemed to be trying to avoid).

GioR Over a year ago

How do you fill the array with the getitem? I thought I would have to use a setter method to fill my cxxTestData array.

DavidW Over a year ago

When you call __getitem__ it returns a PyTestData that shares data with part of the array. Therefore if you change the elements of that PyTestData then it also changes the array.

GioR Over a year ago

I see. However, I my case I need to copy/fill my array with some instatiated pyTestData objects and then pass the array to the solver (processData). I tested your code and it works really nice. I will edit some minor parts only. Please review. Many thanks once again for your valuable advice.

|

Collectives™ on Stack Overflow

cython wrap cpp structs and function with parameter an array of structs

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related