24

I am trying to package together an existing Python code and a new C++ 11 code using CMake and pybind 11. I think I am missing something simple to add into CMake scripts, but can't find it anywhere: pybind11 examples have only C++ code and none of Python, other online resources are rather convoluted and not up-to-date -- so I just can't figure out how to package functions in both languages together and make them available via Python's import my_package down the line... as an example, I have cloned the cmake_example from pybind11 and added a mult function into cmake_example/mult.py

def mult(a, b):
    return a * b

how would I make it visible along with add and subtract to pass the test below? 

import cmake_example as m

assert m.__version__ == '0.0.1'
assert m.add(1, 2) == 3
assert m.subtract(1, 2) == -1
assert m.mult(2, 2) == 4

currently, this test fails..

Thanks!

Improve this question

2 Answers 2

Reset to default
26
+50

The simplest solution has nothing to do with pybind11 as such. What authors usually do when they want to combine pure Python and C/Cython/other native extensions in the same package, is the following.

You create two modules.

  1. mymodule is a public interface, a pure Python module
  2. _mymodule is a private implementation, a complied module

Then in mymodule you import necessary symbols from _mymoudle (and fallback to pure Python version if necessary).

Here's example from yarl package:

  1. quoting.py

    try:
    from ._quoting import _quote, _unquote
    quote = _quote
    unquote = _unquote
except ImportError:  # pragma: no cover
    quote = _py_quote
    unquote = _py_unquote

  2. _quoting.pyx

Update

Here follows the script. For the sake of reproducibility I'm doing it against original cmake_example.

git clone --recursive https://github.com/pybind/cmake_example.git
# at the time of writing https://github.com/pybind/cmake_example/commit/8818f493  
cd cmake_example

Now create pure Python modules (inside cmake_example/cmake_example).

cmake_example/__init__.py

"""Root module of your package"""

cmake_example/math.py

def mul(a, b):
    """Pure Python-only function"""
    return a * b


def add(a, b):
    """Fallback function"""    
    return a + b    

try:
    from ._math import add
except ImportError:
    pass

Now let's modify existing files to turn cmake_example module into cmake_example._math.

src/main.cpp (subtract removed for brevity)

#include <pybind11/pybind11.h>

int add(int i, int j) {
    return i + j;
}

namespace py = pybind11;

PYBIND11_MODULE(_math, m) {
    m.doc() = R"pbdoc(
        Pybind11 example plugin
        -----------------------

        .. currentmodule:: _math

        .. autosummary::
           :toctree: _generate

           add
    )pbdoc";

    m.def("add", &add, R"pbdoc(
        Add two numbers

        Some other explanation about the add function.
    )pbdoc");

#ifdef VERSION_INFO
    m.attr("__version__") = VERSION_INFO;
#else
    m.attr("__version__") = "dev";
#endif
}

CMakeLists.txt

cmake_minimum_required(VERSION 2.8.12)
project(cmake_example)

add_subdirectory(pybind11)
pybind11_add_module(_math src/main.cpp)

setup.py

# the above stays intact

from subprocess import CalledProcessError

kwargs = dict(
    name='cmake_example',
    version='0.0.1',
    author='Dean Moldovan',
    author_email='[email protected]',
    description='A test project using pybind11 and CMake',
    long_description='',
    ext_modules=[CMakeExtension('cmake_example._math')],
    cmdclass=dict(build_ext=CMakeBuild),
    zip_safe=False,
    packages=['cmake_example']
)

# likely there are more exceptions, take a look at yarl example
try:
    setup(**kwargs)        
except CalledProcessError:
    print('Failed to build extension!')
    del kwargs['ext_modules']
    setup(**kwargs)

Now we can build it.

python setup.py bdist_wheel

In my case it produces dist/cmake_example-0.0.1-cp27-cp27mu-linux_x86_64.whl (if C++ compilation fails it's cmake_example-0.0.1-py2-none-any.whl). Here is what it contents (unzip -l ...):

Archive:  cmake_example-0.0.1-cp27-cp27mu-linux_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2017-12-05 21:42   cmake_example/__init__.py
    81088  2017-12-05 21:43   cmake_example/_math.so
      223  2017-12-05 21:46   cmake_example/math.py
       10  2017-12-05 21:48   cmake_example-0.0.1.dist-info/DESCRIPTION.rst
      343  2017-12-05 21:48   cmake_example-0.0.1.dist-info/metadata.json
       14  2017-12-05 21:48   cmake_example-0.0.1.dist-info/top_level.txt
      105  2017-12-05 21:48   cmake_example-0.0.1.dist-info/WHEEL
      226  2017-12-05 21:48   cmake_example-0.0.1.dist-info/METADATA
      766  2017-12-05 21:48   cmake_example-0.0.1.dist-info/RECORD
---------                     -------
    82775                     9 files
Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

thank you, this is a very useful suggestion. i am not proficient enough in python and its packaging toolkit to implement the suggested solution on my own, even for that simple example i put on the github. would you be able to help me by shaping it to have the mult in c++ and a fallback in python, as you have suggested? so I can use it as a starting point for building and distributing my own package? thank you!
@seninp Take a look.
I really like the way this works with keeping a separation between C++ and Python sources and the ability to test/use those independently. Also it builds and tests seamlesly on Travis and codecov is able to pull the coverage metrics. Accepting the answer -- Thank you again!
@seninp Glad it helped. Recently added try-except for setup call so pure Python fallback is actually illustrated (bdist_wheel succeeds without cmake).
@saaj I don't understand how you use the final code after you build. If you're building a library that has both Python and compiled code in a Python module as you suggest, then how do you use it? Do you have to install it first or just put the path in PYTHONPATH to use? If it needs to be installed, are both modules (the compiled and non-compiled) installed so I can import them in Python?
|
2

Once you've cloned the repo, cd to top level directory `cmake_example'

Change ./src/main.cpp to include a "mult" function:

#include <pybind11/pybind11.h>

int add(int i, int j) {
    return i + j;
}

int mult(int i, int j) {
   return i * j;
}

namespace py = pybind11;

PYBIND11_MODULE(cmake_example, m) {
    m.doc() = R"pbdoc(
        Pybind11 example plugin
        -----------------------

        .. currentmodule:: cmake_example

        .. autosummary::
           :toctree: _generate

           add
           subtract
           mult

    )pbdoc";

    m.def("add", &add, R"pbdoc(
        Add two numbers

        Some other explanation about the add function.
    )pbdoc");

   m.def("mult", &mult, R"pbdoc(
        Multiply two numbers

        Some other explanation about the mult function.
    )pbdoc");

(the rest of the file is the same)

Now make it:

$ cmake -H. -Bbuild
$ cmake --build build -- -j3

The module for import will be created in the ./build directory. Go to it, then within a python shell your example should work.

For the namespace import, you could do something with pkgutil:

create the directory structure:

./my_mod
    __init__.py
    cmake_example.***.so

and another parallel structure

./extensions
    /my_mod
        __init__.py
        cmake_example_py.py

and place in ./my_mod/__init__.py

import pkgutil
__path__ = pkgutil.extend_path(__path__, __name__)

from .cmake_example import add, subtract
from .cmake_example_py import mult

in ./extensions/my_mod/__init__.py

from cmake_example_py import mult

Then append both ./my_mod and ./extensions/my_mod to your $PYTHONPATH, it just might work (it does in my example)

Improve this answer

2 Comments

this would be a valid answer if I need to code mult in C language, but I need it to be in Python and packaged using cmake along with add and subtract written in C... thank you
Added someting addressing this using pkgutil. I somehow left out that part in my original post.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.