python use ctypes to pass 2-d array into c function

Question

I want to use C to deal with some computations. For example, I have a C function of adding two matrix:

// mat_add.c
#include <stdlib.h>

void matAdd(int ROW, int COL, int x[][COL], int y[][COL], int z[][COL]){
    int i, j;
    for (i = 0; i < ROW; i++){
        for (j = 0; j < COL; j++){
            z[i][j] = x[i][j] + y[j][j];
        }
    }
}

Then I compiled it into .so file:
gcc -shared -fPIC mat_add.c -o mat_add.so

And in python:

# mat_add_test.py
import ctypes
import numpy as np

def cfunc(x, y):
    nrow, ncol = x.shape
    
    objdll = ctypes.CDLL('./mat_add.so')
    
    func = objdll.matAdd
    func.argtypes = [
        ctypes.c_int,
        ctypes.c_int,
        np.ctypeslib.ndpointer(dtype=np.int, ndim=2, shape=(nrow, ncol)),
        np.ctypeslib.ndpointer(dtype=np.int, ndim=2, shape=(nrow, ncol)),
        np.ctypeslib.ndpointer(dtype=np.int, ndim=2, shape=(nrow, ncol))
    ]
    func_restype = None
    
    z = np.empty_like(x)
    func(nrow, ncol, x, y, z)
    return z


if __name__ == '__main__':
    x = np.array([[1, 2], [3, 4]], dtype=np.int)
    y = np.array([[2, 2], [5, 6]], dtype=np.int)
    z = cfunc(x, y)
    print(z)
    print('end')

Executed this python file, I obtained:

$ python mat_add_test.py 
[[                  3                   4]
 [8386863780988286322 7813586346238636153]]
end

The first row of return matrix is correct, but the second row is wrong. I guess that I don't successfully update the value in z, but I have no idea where the problem is.
Can anyone help? Very thanks!

Your type in C is the wrong length. It works if I try your code with the arrays declared as long. Probably there is some correct procedure with an appropriate typedef - I am not suggesting hard-coding long but it suffices to demonstrate the point. — alani
– alani, Commented Jul 17, 2020 at 10:59
Aside from that, a small bug in your C: y[j][j] should be y[i][j] — alani
– alani, Commented Jul 17, 2020 at 11:01
I change my .c file from int to long, and it works! I forget that np.int is difference to int in c. Very thanks! — Chun-Ye Lu
– Chun-Ye Lu, Commented Jul 17, 2020 at 11:07
You are welcome. To discover the problem I just put a printf statement into the C code - printf("i=%d j=%d x=%d y=%d\n", i, j, x[i][j], y[i][j]); writes i=0 j=0 x=1 y=2 i=0 j=1 x=0 y=0 i=1 j=0 x=2 y=2 i=1 j=1 x=0 y=0. So then it becomes fairly obvious. — alani
– alani, Commented Jul 17, 2020 at 11:09

alani · Accepted Answer · 2020-07-17 12:20:08Z

2

The handling of 2d array in the question is correct. The only problem (apart from a typo in how the C code indexes the y array - y[j][j] should be y[i][j]) is that np.int is np.int64 so this does not correspond to a C int.

To ensure that the types match, an explicit length can be specified in both languages.

In Python: use np.int32 or np.int64 explicitly (instead of np.int).

In C: #include <stdint.h> and then use int32_t or int64_t correspondingly (possibly via a typedef), instead of int.

Then the problem goes away.

For ROW and COL, these are call by value so it is less important (provided of course that the values do not overflow).

What is happening here

In reality a 2d array is still just a 1d sequence of values in memory; the 2 dimensions are just a convenient way to index it.

So in numpy the arrays just before calling C are (in hex):

0000000000000001 0000000000000002 0000000000000003 0000000000000004  <== x
0000000000000002 0000000000000002 0000000000000005 0000000000000006  <== y
UUUUUUUUUUUUUUUU UUUUUUUUUUUUUUUU UUUUUUUUUUUUUUUU UUUUUUUUUUUUUUUU  <== z

where U means undefined / uninitialised data

but in the C code (assuming little endian), treating the arrays as 32-bit, it sees:

inputs
00000001 00000000 00000002 00000000 00000003 00000000 00000004 00000000  <== x
00000002 00000000 00000002 00000000 00000005 00000000 00000006 00000000  <== y
UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU  <== z at start

Then the C code loops over the first 4 elements of each, performing additions, so this produces:

00000003 00000000 00000004 00000000 UUUUUUUU UUUUUUUU UUUUUUUU UUUUUUUU  <== z at end

and back in numpy using a 64-bit int type, now we see:

0000000000000003 0000000000000004 UUUUUUUUUUUUUUUU UUUUUUUUUUUUUUUU  <== output z

Interpreted as a 2-d array, this is array([[3, 4], [whatever, whatever]])

edited Jul 17, 2020 at 12:20

answered Jul 17, 2020 at 11:55

alani

13.2k3 gold badges18 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Chun-Ye Lu Over a year ago

I fully understand why I was wrong and why it appeared some strange numbers. Thank you for your patient explanation!

alani Over a year ago

@Chun-YeLu I wrote the explanation in the answer with wrong lengths -- 16 and 32 bit (4 and 8 hex digits) - now corrected to 32 and 64 bit (8 and 16 hex digits) -- although the principle is of course exactly the same.

Chun-Ye Lu Over a year ago

That's okay ;) I'm not familiar with hexadecimal so I have not noticed about this.

Collectives™ on Stack Overflow

python use ctypes to pass 2-d array into c function

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related