C function called via Python determines wrong array size

Question

I'm using Python to call a .so compiled from C. The C code adds two vectors as follows:

#include <stdio.h>
#include <stdbool.h>

bool add_vectors(const double * const a, const double * const b, double * const c)
{
  if(sizeof(a) != sizeof(b) || sizeof(b) != sizeof(c))
  {
    fprintf(stderr, "Vectors of different lengths cannot be added");
    fprintf(stderr, "Number of elements of a: %d", sizeof(a)/sizeof(double));
    fprintf(stderr, "Number of elements of b: %d", sizeof(b)/sizeof(double));
    fprintf(stderr, "Number of elements of c: %d", sizeof(c)/sizeof(double));
    return false;
  }
  /* Added for diagnostics only; should print 5, does print 1 */
  printf("Size of a: %d\n", sizeof(a)/sizeof(a[0]));
  printf("Size of b: %d\n", sizeof(b)/sizeof(b[0]));
  printf("Size of c: %d\n", sizeof(c)/sizeof(c[0]));

  for(int ii = 0; ii < sizeof(a)/sizeof(double); ++ii)
  {
    c[ii] = a[ii] + b[ii];
  }

  return true;
}

This is compiled in the standard way via

gcc -std=c11 -o add_vectors.so -shared -fPIC add_vectors.c

Now I attempt to call this from the following python code:

#!/usr/bin/env python                                                                    
import ctypes
import numpy
add_vectors_lib = ctypes.cdll.LoadLibrary("add_vectors.so")
add_vectors = add_vectors_lib.add_vectors
add_vectors.retype = ctypes.c_bool

array_1d_double = numpy.ctypeslib.ndpointer(dtype = numpy.double, ndim=1, flags="C_CONTIGUOUS")
add_vectors.argtypes = [array_1d_double, array_1d_double, array_1d_double]

#Random vectors to add:
a = numpy.double([1,2,3,4,5])
b = numpy.double([3,4,5,6,7])
#Zero out the return value:
c = numpy.double([0,0,0,0,0])
add_vectors(a, b,c)
print(a)
print(b)
print(c)

But the output is:

Size of a: 1
Size of b: 1
Size of c: 1
[ 1.  2.  3.  4.  5.]
[ 3.  4.  5.  6.  7.]
[ 4.  0.  0.  0.  0.]

How do I make the C code recognize the proper size of these arrays and/or make the Python pass "knowledge" of the array size to the C code?

C code is incorrect. sizeof(double*) is not size of array, this is just pointer size. You need additional parameter with vectors size. — Alex F
– Alex F, Commented Dec 24, 2014 at 12:09
Goddammit, you're right. Arrays passed as parameters to functions only "know" the pointer size. — user14717
– user14717, Commented Dec 24, 2014 at 12:17

Manos · Accepted Answer · 2014-12-24 12:23:28Z

4

sizeof() is a compile time operator. In the case of a pointer returns the actual size of the pointer itself. Usually, this is 4 bytes in case of 32-bit architecture and 8 in 64-bit respectively.

In the case that you passed the actual variable of a statically allocated array, it would return the total size of the array in bytes.

answered Dec 24, 2014 at 12:23

Manos

2,18617 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

D3Hunter · Accepted Answer · 2014-12-24 13:54:03Z

The quite newbee problem of sizeof() has already pointed out in comment.

Well, in order to answer your question How do I make the C code recognize the proper size of these arrays and/or make the Python pass "knowledge" of the array size to the C code. I tried to learn how to write one module with C in python by following this tutorial (I'm interested in learning python).

Notice: it's a quite long answer, omit the code part as your wish.

Your way of writing module is complex and bug prone. You need a wrapper of add_vectors which takes PyObject *args as argument, so you can check the type of your parameters(with PyArg_ParseTuple) and number of elements(with PyArray_DIM) in the array correctly.

This is part of my code:

add_vectors.c

#include <stdio.h>
#include <stdbool.h>

void add_vectors(const double * const a, const double * const b,
                 double * const c, int len)
{
    int ii;
    for(ii = 0; ii < len; ++ii)
    {
        c[ii] = a[ii] + b[ii];
    }
}

_add_vectors.c

#include <Python.h>
#include <numpy/arrayobject.h>

void add_vectors(const double * const a, const double * const b,
                 double * const c, int len);

static PyObject *add_vectors_wrapper(PyObject *self, PyObject *args);

static PyMethodDef module_methods[] = {
    {"add_vectors", add_vectors_wrapper, METH_VARARGS, NULL},
    {NULL, NULL, 0, NULL}
};

PyMODINIT_FUNC init_add_vectors(void)
{
    PyObject *m = Py_InitModule3("_add_vectors", module_methods,
                                 NULL);
    if (m == NULL)
        return;

    import_array();
}
static PyObject *add_vectors_wrapper(PyObject *self, PyObject *args)
{
    PyObject *x_obj, *y_obj, *z_obj;

    if (!PyArg_ParseTuple(args, "OOO", &x_obj, &y_obj,
                          &z_obj))
        return NULL;

    /* Interpret the input objects as numpy arrays. */
    PyObject *x_array = PyArray_FROM_OTF(x_obj, NPY_DOUBLE, NPY_IN_ARRAY);
    PyObject *y_array = PyArray_FROM_OTF(y_obj, NPY_DOUBLE, NPY_IN_ARRAY);
    PyObject *z_array = PyArray_FROM_OTF(z_obj, NPY_DOUBLE, NPY_IN_ARRAY);

    /* If that didn't work, throw an exception. */
    if (x_array == NULL || y_array == NULL || z_array == NULL) {
        Py_XDECREF(x_array);
        Py_XDECREF(y_array);
        Py_XDECREF(z_array);
        return NULL;
    }

    /* How many data points are there? */
    int xN = (int)PyArray_DIM(x_array, 0);
    int yN = (int)PyArray_DIM(y_array, 0);
    int zN = (int)PyArray_DIM(z_array, 0);

    /* size check */
    if (xN != yN || yN != zN) {
        fprintf(stderr, "Vectors of different lengths cannot be added\n");
        fprintf(stderr, "Number of elements of a: %d\n", xN);
        fprintf(stderr, "Number of elements of b: %d\n", yN);
        fprintf(stderr, "Number of elements of c: %d\n", zN);
        PyObject *ret = Py_BuildValue("s", "Failed");
        return ret;
    }

    double *x = (double*)PyArray_DATA(x_array);
    double *y = (double*)PyArray_DATA(y_array);
    double *z = (double*)PyArray_DATA(z_array);

    add_vectors(x, y, z, xN);

    /* Clean up. */
    Py_DECREF(x_array);
    Py_DECREF(y_array);
    Py_DECREF(z_array);

    /* Build the output tuple */
    PyObject *ret = Py_BuildValue("s", "Success");
    return ret;
}

setup.py (Run with./setup.py build_ext --inplace)

#!/usr/bin/env python

from distutils.core import setup, Extension
import numpy.distutils.misc_util

setup(
    ext_modules=[Extension("_add_vectors", 
                           ["_add_vectors.c", "add_vectors.c"])],
    include_dirs=numpy.distutils.misc_util.get_numpy_include_dirs(),
)

addnum.py ( a simple testcase)

#!/usr/bin/env python                                                                    
import ctypes
import numpy
from _add_vectors import add_vectors

#Random vectors to add:
a = numpy.double([1,2,3,4])
b = numpy.double([3,4,5,6,7])

#Zero out the return value:
c = numpy.double([0,0,0,0,0])
add_vectors(a, b, c)
print(a)
print(b)
print(c)

result

ubuntu-user:python-module$ ./addnum.py 
[ 1.  2.  3.  4.  5.]
[ 3.  4.  5.  6.  7.]
[  4.   6.   8.  10.  12.]

The full extension module is cool (and thanks for a good example), I often prefer to use just add_vectors.c and ctypes to call into it depending on give much c etc I want to write. So many good ways, gotta love python.
This is a good answer, but it seems to make my C library Python-callable only, which I don't want to do. Is this impression correct?
I tried that. Result _add_vectors.so have some external symbols, like PyExc_RuntimeError(it's glabal variable, so lazy loading doesn't help). I define them in my code to resolve symbol lookup error when loading, but that's ugly and may cause problems. So, yeah, it's better be Python-callable only (previous trick do be able to call add_vectors in any C program, but not suggested).

Collectives™ on Stack Overflow

C function called via Python determines wrong array size

2 Answers 2

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related