pybind11: Python callback executed in C++ with parameter modification

Question

I'm working on Python bindings of my C++ library (a mathematical optimization solver) and I'm stuck at a point where I create a Python callback evaluate_constraints() that takes two arguments, pass it to the C++ library and evaluate it with C++ arguments. The callback modifies its second parameter constraints based on its first parameter x.

// C++ code
#include "Vector.hpp"
#include <pybind11/pybind11.h>
namespace py = pybind11;

void solve(const std::function<void(const Vector&, Vector&)>& evaluate_constraints) {
  const Vector x = ...;
  Vector constraints = ...;
  evaluate_constraints(x, constraints);
}

PYBIND11_MODULE(myCppModule, module) {
   py::class_<Vector>(module, "Vector")
      .def(py::init<size_t>(), "Constructor")
      .def("__getitem__", [](const Vector& vector, size_t index) {
         return vector[index];
      })
      .def("__setitem__", [](Vector& vector, size_t index, double value) {
         vector[index] = value;
      });
   module.def("solve", &solve);
}

# Python code
import myCppModule

def evaluate_constraints(x, constraints):
  constraints[0] = function of x
  constraints[1] = function of x
  ...

myCppModule.solve(evaluate_constraints)

Unfortunately, some copy must happen somewhere, because the C++ object constraints is not modified. I'm not sure whether I missed something totally obvious (I've stumbled upon suggestions to use py::return_value_policy::reference_internal, but to no avail) or whether it is indeed a bit tricky to address. Hope you can crack it!

Note: the second parameter is a Vector here, but for other callbacks, it could be a C++ matrix type.

Ahmed AEK · Accepted Answer · 2025-07-13 11:16:24Z

pybind11 stores the C++ object by value in the python object, think of the python object as a struct whose first member is a control block and the second member is your C++ class, so in order to construct the python object your C++ object has to be copied.

pybind11 also allows you to store shared_ptr inside your python object .... this shared_ptr can have an empty deleter in case you don't want python to manage the lifetime of your object, but it will still allocate the control block which is wasteful, and is very unsafe as the user can hold onto your C++ object and cause UB, so from a safety point of view, just allocate your object with make_shared and let python control its lifetime.

another solution if you want to reinvent numpy is to create a "reference-like" object, call it VectorRef that only stores a pointer to your Vector similar to a span and wrap that instead, but be very careful of lifetimes, remember python is expecting to own this object and can hold onto it. it will also help to create a VectorCRef that provides a const view to avoid any copies.

numpy can be used to wrap your matrix and can provide both a read-only and a read-write views over the data using the buffer protocol, but the code shown will only create the buffer from the python object, which requires copying the C++ object into the python object, and you cannot store it on the stack, so to avoid this copy you'll have to manually create a numpy array if you want it to point to stack memory and make your code less safe ... or just give up a create the python object by casting to a py::object then read and write from the copy.

you can also just use Eigen instead which has out-of-the-box integration but is subject to a similar limitation requiring Eigen::Ref to be used to pass matricies by reference, which is just the VectorRef i described above.

Thanks a lot for these suggestions! I went for the "reference-like" object and it seems to work great :)

Collectives™ on Stack Overflow

pybind11: Python callback executed in C++ with parameter modification

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related