pybind11 modify numpy array from C++

Question

EDIT: It works now, I do not know why. Don't think I changed anything

I want to pass in and modify a large numpy array with pybind11. Because it's large I want to avoid copying it and returning a new one.

Here's the code:

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
#include <pybind11/numpy.h>
#include <vector>

// C++ code
void calc_sum_cost(float* ptr, int N, int M, float* ptr_cost) {
  for(int32_t i = 1; i < N; i++) {
    for(int32_t j = 1; j < M; j++) {
      float upc = ptr[(i-1) * M + j];
      float leftc = ptr[i * M + j - 1];
      float diagc = ptr[(i-1) * M + j - 1];
      float transition_cost = std::min(upc, std::min(leftc, diagc));
      if (transition_cost == diagc) {
        transition_cost += 2 * ptr_cost[i*M + j];
      } else {
        transition_cost += ptr_cost[i*M + j];
      }
      std::cout << transition_cost << std::endl;
      ptr[i * M + j] = transition_cost;
    }
  }
}

// Interface

namespace py = pybind11;

// wrap C++ function with NumPy array IO
py::object wrapper(py::array_t<float> array,
                  py::array_t<float> arrayb) {
  // check input dimensions
  if ( array.ndim()     != 2 )
    throw std::runtime_error("Input should be 2-D NumPy array");

  auto buf = array.request();
  auto buf2 = arrayb.request();
  if (buf.size != buf2.size) throw std::runtime_error("sizes do not match!");

  int N = array.shape()[0], M = array.shape()[1];

  float* ptr = (float*) buf.ptr;
  float* ptr_cost = (float*) buf2.ptr;
  // call pure C++ function
  calc_sum_cost(ptr, N, M, ptr_cost);
  return py::cast<py::none>(Py_None);
}

PYBIND11_MODULE(fast,m) {
  m.doc() = "pybind11 plugin";
  m.def("calc_sum_cost", &wrapper, "Calculate the length of an array of vectors");
}

I think the py::array::forcecast is causing a conversion and so leaving the input matrix unmodified (in python). When I remove that though I get a runtime error, when I remove ::c_style it runs but again in python the numpy array is the same.

Basically my question is how can one pass and modify a numpy array with pybind11?

Naively I would expect that to have to pass your function arguments by reference...?! I.e. to use py::array_t<float>& array. — Tom de Geus
– Tom de Geus, Commented Feb 22, 2019 at 17:16
The arguments do not need to be passed by reference because the py::array types internally share the same buffer if possible i.e. when the element types match. — Saul
– Saul, Commented Sep 27, 2019 at 10:08

Saul · Accepted Answer · 2019-03-07 07:17:50Z

15

I just had the same problem. If, from Python, you pass a numpy array of the type matching the C++ argument then no conversion happens, and you can modify the data in-place i.e. for py::array_t<float> argument pass in a numpy np.float32 array. If you happen to pass in a np.float64 array (the default type) then pybind11 does the conversion due to the py::array::forcecast template parameter (default on py::array_t<T>), so your C++ function only gets a converted copy of a numpy array, and any changes are lost after returning.

answered Mar 7, 2019 at 7:17

Saul

4306 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

zhiqiu Over a year ago

Will forcecast does copy if I pass numpy.array slice to C++?

Saul Over a year ago

@zhiqiu, I haven't tried that, but I believe a slice would still be passed as a shared underlying buffer if the element types match i.e. without copying. The py:array shares the same buffer if it can, and no need to accept it by reference in C++ either.

Julia Meshcheryakova · Accepted Answer · 2023-01-01 13:27:31Z

0

You should define the wrapper function to pass parameters by reference:

py::object wrapper(py::array_t<float> &array,
                  py::array_t<float> &arrayb)

And just as Saul's answer shows, you should pass numpy array with type of np.float32 to match the C++ array type, otherwise you can not change the passed arrays from Python in C++ function.

edited Jan 1, 2023 at 13:27

Julia Meshcheryakova

3,3023 gold badges24 silver badges44 bronze badges

answered Jan 1, 2023 at 7:59

JasVixBan

1

Collectives™ on Stack Overflow

pybind11 modify numpy array from C++

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related