46

I am going to send a c++ array to a python function as numpy array and get back another numpy array. After consulting with numpy documentation and some other threads and tweaking the code, finally the code is working but I would like to know if this code is written optimally considering the:

  • Unnecessary copying of the array between c++ and numpy (python).
  • Correct dereferencing of the variables.
  • Easy straight-forward approach.

C++ code:

// python_embed.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"

#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include "Python.h"
#include "numpy/arrayobject.h"
#include<iostream>

using namespace std;

int _tmain(int argc, _TCHAR* argv[])
{
    Py_SetProgramName(argv[0]);
    Py_Initialize();
    import_array()

    // Build the 2D array
    PyObject *pArgs, *pReturn, *pModule, *pFunc;
    PyArrayObject *np_ret, *np_arg;
    const int SIZE{ 10 };
    npy_intp dims[2]{SIZE, SIZE};
    const int ND{ 2 };
    long double(*c_arr)[SIZE]{ new long double[SIZE][SIZE] };
    long double* c_out;
    for (int i{}; i < SIZE; i++)
        for (int j{}; j < SIZE; j++)
            c_arr[i][j] = i * SIZE + j;

    np_arg = reinterpret_cast<PyArrayObject*>(PyArray_SimpleNewFromData(ND, dims, NPY_LONGDOUBLE, 
        reinterpret_cast<void*>(c_arr)));

    // Calling array_tutorial from mymodule
    PyObject *pName = PyUnicode_FromString("mymodule");
    pModule = PyImport_Import(pName);
    Py_DECREF(pName);
    if (!pModule){
        cout << "mymodule can not be imported" << endl;
        Py_DECREF(np_arg);
        delete[] c_arr;
        return 1;
    }
    pFunc = PyObject_GetAttrString(pModule, "array_tutorial");
    if (!pFunc || !PyCallable_Check(pFunc)){
        Py_DECREF(pModule);
        Py_XDECREF(pFunc);
        Py_DECREF(np_arg);
        delete[] c_arr;
        cout << "array_tutorial is null or not callable" << endl;
        return 1;
    }
    pArgs = PyTuple_New(1);
    PyTuple_SetItem(pArgs, 0, reinterpret_cast<PyObject*>(np_arg));
    pReturn = PyObject_CallObject(pFunc, pArgs);
    np_ret = reinterpret_cast<PyArrayObject*>(pReturn);
    if (PyArray_NDIM(np_ret) != ND - 1){ // row[0] is returned
        cout << "Function returned with wrong dimension" << endl;
        Py_DECREF(pFunc);
        Py_DECREF(pModule);
        Py_DECREF(np_arg);
        Py_DECREF(np_ret);
        delete[] c_arr;
        return 1;
    }
    int len{ PyArray_SHAPE(np_ret)[0] };
    c_out = reinterpret_cast<long double*>(PyArray_DATA(np_ret));
    cout << "Printing output array" << endl;
    for (int i{}; i < len; i++)
        cout << c_out[i] << ' ';
    cout << endl;

    // Finalizing
    Py_DECREF(pFunc);
    Py_DECREF(pModule);
    Py_DECREF(np_arg);
    Py_DECREF(np_ret);
    delete[] c_arr;
    Py_Finalize();
    return 0;
}

In CodeReview, there is a fantastic answer: Link...

Improve this question
8
  • have you looked at Boost.Numpy? See this very simple example: github.com/ndarray/Boost.NumPy/blob/master/libs/numpy/example/… Commented May 26, 2015 at 4:22
  • or using Cython? stackoverflow.com/a/18176741/2230844 Commented May 26, 2015 at 4:25
  • @denfromufa, I don't want to use boost here, also cython is not an option as I am extending C++ using python/numpy. Commented May 26, 2015 at 7:14
  • Did you look at SWIG (docs.scipy.org/doc/numpy/reference/swig.interface-file.html)? It allows to have memory views in numpy on C++-arrays and vice versa. It takes care of the wrapping automatically. Though I have never used it in the C++ to Python direction. Commented May 30, 2015 at 8:11
  • 1
    @Dietrich, Thanks, SWIG is also good, but here I would like to work purely with Python/C++ API. I appreciate any comments on the provided code. Commented May 30, 2015 at 9:22

4 Answers 4

Reset to default
9

Try out xtensor and the xtensor-python python bindings.

xtensor is a C++ library meant for numerical analysis with multi-dimensional array expressions.

xtensor provides

  • an extensible expression system enabling numpy-style broadcasting (see the numpy to xtensor cheat sheet).
  • an API following the idioms of the C++ standard library.
  • tools to manipulate array expressions and build upon xtensor.
  • bindings for Python, but also R and Julia.

Example of usage

Initialize a 2-D array and compute the sum of one of its rows and a 1-D array.

#include <iostream>
#include "xtensor/xarray.hpp"
#include "xtensor/xio.hpp"

xt::xarray<double> arr1
  {{1.0, 2.0, 3.0},
   {2.0, 5.0, 7.0},
   {2.0, 5.0, 7.0}};

xt::xarray<double> arr2
  {5.0, 6.0, 7.0};

xt::xarray<double> res = xt::view(arr1, 1) + arr2;

std::cout << res;

Outputs

{7, 11, 14}

Creating a Numpy-style universal function in C++.

#include "pybind11/pybind11.h"
#include "xtensor-python/pyvectorize.hpp"
#include <numeric>
#include <cmath>

namespace py = pybind11;

double scalar_func(double i, double j)
{
    return std::sin(i) - std::cos(j);
}

PYBIND11_PLUGIN(xtensor_python_test)
{
    py::module m("xtensor_python_test", "Test module for xtensor python bindings");

    m.def("vectorized_func", xt::pyvectorize(scalar_func), "");

    return m.ptr();
}

Python code:

import numpy as np
import xtensor_python_test as xt

x = np.arange(15).reshape(3, 5)
y = [1, 2, 3, 4, 5]
z = xt.vectorized_func(x, y)
z

Outputs

[[-0.540302,  1.257618,  1.89929 ,  0.794764, -1.040465],
 [-1.499227,  0.136731,  1.646979,  1.643002,  0.128456],
 [-1.084323, -0.583843,  0.45342 ,  1.073811,  0.706945]]
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

seems interesting, but you are extending python using C++, I need to extend C++ using python.
Ok this actually something that is done in the tests for xtensor where we instanciate a python interpreter from c++ and start creating arrays.
8

We will be passing 2D array to python function written in file pyCode.py:

def pyArray (a):
    print ("Contents of a :")
    print (a)
    c = 0
    return c
  1. For C++ to Python: File: c_code.cpp
#include <Python.h>
#include <stdio.h>
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <numpy/arrayobject.h>

float Array [] = {1.2, 3.4, 5.6, 7.8};

int main (int argc, char *argv[])
{
    float *ptr = Array;
    PyObject *pName, *pModule, *pDict, *pFunc, *pArgs;
    npy_intp dims[1] = { 4 };
    PyObject *py_array;

    setenv("PYTHONPATH",".",1);
    Py_Initialize ();
    pName = PyUnicode_FromString ("pyCode");

    pModule = PyImport_Import(pName);

    pDict = PyModule_GetDict(pModule);

    import_array ();                                   

    py_array = PyArray_SimpleNewFromData(1, dims, NPY_FLOAT, ptr);
    

    pArgs = PyTuple_New (1);
    PyTuple_SetItem (pArgs, 0, py_array);

    pFunc = PyDict_GetItemString (pDict, (char*)"pyArray"); 

    if (PyCallable_Check (pFunc))
    {
        PyObject_CallObject(pFunc, pArgs);
    } else
    {
        cout << "Function is not callable !" << endl;
    }

    Py_DECREF(pName);
    Py_DECREF (py_array);                             
    Py_DECREF (pModule);
    Py_DECREF (pDict);
    Py_DECREF (pFunc);

    Py_Finalize ();                                    

    return 0;
}

compile the code: g++ -g -fPIC c_code.cpp -o runMe -lpython3.5m -I/usr/include/python3.5m/

  1. From OpenCV Mat to Python:

file: cv_mat_code.cpp

#include <iostream>
#include <Python.h>
#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
#include <numpy/arrayobject.h>

#include <opencv2/opencv.hpp>

using namespace cv;
using namespace std;

int main (int argc, char *argv[])
{
    float data[10] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

    Mat mat1 (cv::Size (5, 2), CV_32F, data, Mat::AUTO_STEP);
    int row = 0;
    float *p = mat1.ptr<float>(row);

    cout << "Mat" << mat1 <<endl;

    PyObject *pName, *pModule, *pDict, *pFunc, *pArgs;
    npy_intp dims[2] = { 2, 5 };
    PyObject *py_array;

    setenv("PYTHONPATH",".",1);
    Py_Initialize ();
    pName = PyUnicode_FromString ("pyCode");
    
    pModule = PyImport_Import(pName);

    pDict = PyModule_GetDict(pModule);

    // Required for the C-API : http://docs.scipy.org/doc/numpy/reference/c-api.array.html#importing-the-api
    import_array ();

    py_array = PyArray_SimpleNewFromData(2, dims, NPY_FLOAT, p);

    pArgs = PyTuple_New (1);
    PyTuple_SetItem (pArgs, 0, py_array);

    pFunc = PyDict_GetItemString (pDict, (char*)"pyArray"); 

    if (PyCallable_Check (pFunc))
    {
        PyObject_CallObject(pFunc, pArgs);
    } else
    {
        cout << "Function is not callable !" << endl;
    }

    Py_DECREF(pName);
    Py_DECREF (py_array);                             
    Py_DECREF (pModule);
    Py_DECREF (pDict);
    Py_DECREF (pFunc);

    Py_Finalize ();                                  

    return 0;
}

Compile the code: g++ -g -fPIC cv_mat_code.cpp -o runMe -lpython3.5m -I/usr/include/python3.5m/ -I/usr/include/ -lopencv_core -lopencv_imgproc -lopencv_highgui

Improve this answer

7 Comments

why do you have 42 float in your data array but assign only 10 elements in your Mat?
@simplename Thanks! made the changes.
The PyObject_callObject gets blocked if there's a cv2 call inside there. Is this a known issue with opencv for python 2.7
its called python code embedding and hence they are blocking. To understand how main process invokes interpreter here.
I misspoke, what I meant to say is it hangs at the cv2 call and doesn't proceed.
|
5

As an additional way, without touching directly to the Python C API, it is possible to use pybind11 ( header-only library) :

CPP :

#include <pybind11/embed.h> // everything needed for embedding
#include <iostream>
#include <Eigen/Dense>  
#include<pybind11/eigen.h>
using Eigen::MatrixXd;
namespace py = pybind11;

int main() 
{    
  try 
  {          
        Py_SetProgramName("PYTHON");
        py::scoped_interpreter guard{}; 

        py::module py_test = py::module::import("py_test");

        MatrixXd m(2,2);
        m(0,0) = 1;
        m(1,0) = 2;
        m(0,1) = 3;
        m(1,1) = 4;

        py::object result = py_test.attr("test_mat")(m);

        MatrixXd res = result.cast<MatrixXd>();
        std::cout << "In c++ \n" << res << std::endl;
  }
  catch (std::exception ex)
  {
      std::cout << "ERROR   : " << ex.what() << std::endl;
  }
  return 1;
}

In py_test.py :

def test_mat(m):
    print ("Inside python m = \n ",m )
    m[0,0] = 10
    m[1,1] = 99 
    return m

Output :

Inside python m =
  [[ 1.  3.]
  [ 2.  4.]]
In c++
10  3
 2 99

See the official documentation.

ps: I'm using Eigen for the C++ Matrix.

Improve this answer

1 Comment

Thanks for the great example; I've just replicated it. Found that this line needed to be: std::cout << "In c++ \n" << res << std::endl;
4

From my experience that seems to be pretty efficient. To get even more efficiency out of it try this : http://ubuntuforums.org/showthread.php?t=1266059

Using weave you can inline C/C++ code in Python so that could be useful.

http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.weave.inline.html

Here's a link on how Python can be used to interface between many different languages along with examples.

http://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html

This is a quick and easy example of how to pass numpy arrays to c++ using Cython:

http://www.birving.com/blog/2014/05/13/passing-numpy-arrays-between-python-and/

Improve this answer

5 Comments

I am extending c++ using python not the opposite so cython, weave are not relevant. Only your first link might be relevant. I appreciate if you could put any possible improvement to the provided code though.
Cython is still relevant as you can call it from C++/C by using the public keyword.
@BeRecursive, yes it could also work. Now I am mainly concerned about the performance of my code given in the question.
If you want to improve the speed of the Python bit I can also suggest using pypy.org Using PyPy increases the speed drastically.
Weave is pretty much deprecated in favor of Cython now. The standalone weave package recommends using it only if you have existing code that depends on weave which has not yet been ported.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.