Cython how to copy from a numpy buffer to C struct using memcpy?

Question

This example compiles and runs in Jupyter notebook and demonstrates the issue.

I'm having an issue this line,

    memcpy(line.points, <point_t *> temp.data, sizeof(point_dtype) * n)

which I expect copies a buffer from numpy array temp to a points buffer in the C struct line.

I expect the value of points to be 1.0 but they are junk, basically uninitialised memory. What am I doing wrong!?

%%cython -a

import numpy as np
cimport numpy as cnp
from libc.stdlib cimport malloc
from libc.string cimport memcpy

cdef struct point:
    cnp.float64_t x
    cnp.float64_t y
ctypedef point point_t


cdef struct line:
    point_t * points
ctypedef line line_t


point_dtype = np.dtype([
    ("x", np.float64),
    ("y", np.float64)
])

cdef line_t * make_line():
    """ Make a line of 3 points. Internally does the creation using Numpy 
        and memcpy the result to the line_t C struct.
    """

    # The number of points in the array
    n = 3
    
    cdef cnp.ndarray points = np.empty(n, dtype=point_dtype)
    points["x"] = 0.0
    points["y"] = 0.0

    # Dynamically allocate a line C struct
    line = <line_t*> malloc( sizeof(line_t) )

    # Dynamically allocate space for "n" points
    line.points = <point_t*> malloc( sizeof(point_t) * n)

    # In this toy example we will modify "points" in a temporary array
    # this is closer to what I'm trying to achieve in my actual code.
    temp = np.empty(n, dtype=point_dtype)
    temp[:] = points[:]
    temp["x"] += 1.0
    temp["y"] += 1.0

    # Memory copy from the array's buffer into the struct
    memcpy(line.points, <point_t *> temp.data, sizeof(point_dtype) * n)

    print(line.points[0])
    # {'x': 5e-324, 'y': 4.6451830626356e-310}
    # 
    # !!!! Expected !!!!
    # {'x': 1.0, 'y': 1.0}
    
    # Assert fails
    assert line.points[0].x == 1.0
    assert line.points[0].y == 1.0


def test_create_line():
    make_line()

DavidW · Accepted Answer · 2021-02-27 19:03:01Z

2

The actual bug is:

cdef cnp.ndarray temp = np.empty(n, dtype=point_dtype)

without the cdef cnp.ndarray then temp.data is some kind of Python object (not sure to what exactly), which you then cast as a point_t* and so copying from it fails.

I think in principle you should define Point as cdef packed struct Point since I think Numpy data is packed internally. In this case I don't think it makes a difference.

It might be better to use memoryviews, and then you can assert the C-contiguousness of your arrays.

cdef point_t[::1] temp_view = temp

# Memory copy from the array's buffer into the struct
memcpy(line.points, &temp_view[0], sizeof(point_dtype) * n)

In this case I wouldn't bother typing temp and points because there's really no advantage to it. The nice thing about the memoryview approach is that it avoids a cast (and the cast helped mask your real error) and it includes some checks that your assumptions about data-size/layout and contiguousness are correct.

answered Feb 27, 2021 at 19:03

DavidW

31.2k7 gold badges64 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Daniel Farrell Over a year ago

Thank you so so much! I was stumped for hours with this. Two quick questions: 1) would you mind explaining the &temp_view[0] part, is it "take the address of the first value in the memory view", 2) I just checked the cython docs, but it seems there aren't options when compiling which would have helped catch this, something like, i.e. a Cython option like "require types"

DavidW Over a year ago

1) it is indeed "take the address of the first value in the memory view". 2) I don't think there's a compiler option to require everything to be typed. Part of the problem is that any time you use a <> cast you're saying "I know better", and if you're wrong then all bets are off.

Collectives™ on Stack Overflow

Cython how to copy from a numpy buffer to C struct using memcpy?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related