21

I am trying to call a python function from a C++ code which contains main() function using Pybind11. But I found very few references are available. Most of existing documents talk about the reversed direction, i.e. calling C++ from Python.

Is there any complete example showing how to do that? The only reference I found is: https://github.com/pybind/pybind11/issues/30

But it has very little information.

3 Answers 3

31

The answer to your question really has two parts: one about calling a Python function from C++, the other about embedding the interpreter.

Calling a function in pybind11 is simply a matter of getting that function into a pybind11::object variable, on which you can invoke operator() to attempt to call the object. (It doesn't have to be a function, but just something callable: for example, it could also be an object with a __call__ method). For example, to call math.sqrt(2) from C++ code you'd use:

auto math = py::module::import("math");
auto resultobj = math.attr("sqrt")(2);
double result = resultobj.cast<double>();

or you could condense it all to just:

double result = py::module::import("math").attr("sqrt")(2).cast<double>();

The second part of the question involves how to do this from a C++ executable. When building an executable (i.e. when your C++ code contains main()) you have to embed the Python interpreter in your binary before you can do anything with Python (like calling a Python function).

Embedded support is a new feature added in the current pybind11 master branch (which will become the 2.2 release). Here's a basic example that starts an embedded Python interpreter and calls a Python function (math.sqrt):

#include <pybind11/embed.h>
#include <iostream>

namespace py = pybind11;

int main() {
    py::scoped_interpreter python;

    auto math = py::module::import("math");
    double root_two = math.attr("sqrt")(2.0).cast<double>();

    std::cout << "The square root of 2 is: " << root_two << "\n";
}

Outputs:

The square root of 2 is: 1.41421

More examples and documentation of calling functions and embedding are available at http://pybind11.readthedocs.io/en/stable/advanced/pycpp/object.html and http://pybind11.readthedocs.io/en/stable/advanced/embedding.html, respectively.

Sign up to request clarification or add additional context in comments.

1 Comment

The links above are broken. Use the one below: - pybind11.readthedocs.io/en/stable/advanced/embedding.html
21

Jasons answer is pretty much on point, but I want to add a slightly more complex (and clean) example calling a python method with a numpy input. I want to showcase two points:

  1. We can cast a py::object to a py::function using py::reinterpret_borrow<py::function>
  2. We can input a std::vector that automatically gets converted to a numpy.array

Note that the user is responsible for making sure that the PyModule.attr is actually a python function. Also note that the type conversion works for a wide variety of c++ types (see here for details).

In this example I want to use the method scipy.optimize.minimize with a starting point x0 that is provided from the c++ interface.

#include <iostream>
#include <vector>
#include <pybind11/pybind11.h>
#include <pybind11/embed.h>  // python interpreter
#include <pybind11/stl.h>  // type conversion

namespace py = pybind11;

int main() {
  std::cout << "Starting pybind" << std::endl;
  py::scoped_interpreter guard{}; // start interpreter, dies when out of scope

  py::function min_rosen =
      py::reinterpret_borrow<py::function>(   // cast from 'object' to 'function - use `borrow` (copy) or `steal` (move)
          py::module::import("py_src.exec_numpy").attr("min_rosen")  // import method "min_rosen" from python "module"
      );

  py::object result = min_rosen(std::vector<double>{1,2,3,4,5});  // automatic conversion from `std::vector` to `numpy.array`, imported in `pybind11/stl.h`
  bool success = result.attr("success").cast<bool>();
  int num_iters = result.attr("nit").cast<int>();
  double obj_value = result.attr("fun").cast<double>();
}

with the python script py_src/exec_numpy.py

import numpy as np
from scipy.optimize import minimize, rosen, rosen_der

def min_rosen(x0):
    res = minimize(rosen, x0)
    return res

Hope this helps someone!

6 Comments

Hi , I am trying to use your code but I get i am getting an expetion : Microsoft C++ exception: pybind11::error_already_set at memory
The last three lines of the C++ code have res.attr but these should be result.attr.
Is there a way to avoid copying the data from the STL container into a numpy.array during the automatic conversion? Something like creating a Python object that points to the memory of the STL container...
I tried to find a way to avoid copies from STL to Python but couldn't find one.
@Rackbox You can avoid copying data using the buffer protocol. I don't know if std::vector has the buffer protocol already built, or if you have to do it yourself. But the pybind11 test scripts do tests for the buffer protocol so you should be able to look there for how to use it.
|
7
  1. project structure
  • CMakeLists.txt
  • calc.py
  • main.cpp
  1. main.cpp

    #include <pybind11/embed.h>
    #include <iostream>
    namespace py = pybind11;
    using namespace py::literals;
    
    
    int main() {
        py::scoped_interpreter guard{};
    
        // append source dir to sys.path, and python interpreter would find your custom python file
        py::module_ sys = py::module_::import("sys");
        py::list path = sys.attr("path");
        path.attr("append")("..");
    
        // import custom python class and call it
        py::module_ tokenize = py::module_::import("calc");
        py::type customTokenizerClass = tokenize.attr("CustomTokenizer");
        py::object customTokenizer = customTokenizerClass("/Users/Caleb/Desktop/codes/ptms/bert-base");
        py::object res = customTokenizer.attr("custom_tokenize")("good luck");
    
        // show the result
        py::list input_ids = res.attr("input_ids");
        py::list token_type_ids = res.attr("token_type_ids");
        py::list attention_mask = res.attr("attention_mask");
        py::list offsets = res.attr("offset_mapping");
        std::string message = "input ids is {},\noffsets is {}"_s.format(input_ids, offsets);
        std::cout << message << std::endl;
    }
    
  2. calc.py

    from transformers import BertTokenizerFast
    
    
    class CustomTokenizer(object):
        def __init__(self, vocab_dir):
            self._tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
    
        def custom_tokenize(self, text):
            return self._tokenizer(text, return_offsets_mapping=True)
    
    
    def build_tokenizer(vocab_dir: str) -> BertTokenizerFast:
        tokenizer = BertTokenizerFast.from_pretrained(vocab_dir)
        return tokenizer
    
    
    def tokenize_text(tokenizer: BertTokenizerFast, text: str) -> dict:
        res = tokenizer(text, return_offsets_mapping=True)
        return dict(res)
    
    
  3. CMakeLists.txt

    cmake_minimum_required(VERSION 3.4)
    project(example)
    set(CMAKE_CXX_STANDARD 11)
    
    # set pybind11 dir
    set(pybind11_DIR /Users/Caleb/Softwares/pybind11)
    find_package(pybind11 REQUIRED)
    
    # set custom python interpreter(under macos)
    link_libraries(/Users/Caleb/miniforge3/envs/py38/lib/libpython3.8.dylib)
    
    add_executable(example main.cpp)
    target_link_libraries(example PRIVATE pybind11::embed)
    

2 Comments

An exception is throwing in line py::module_ tokenize = py::module_::import("calc");
transformers package is needed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.