Embedding python with pybind11. Virtual environment doesn't work

Question

I am trying to make a simple c++ application which translates phrases using googletrans python library. So I've chosen pybind11 for this purpose to embed python. Also I use cmake for code configuration.

Everything works just fine when I'm using the global python installation, but I don't understand how to use virtual environment with pybind and the whole process of setting the correct python interpreter, paths etc. in cmake much.

I've found this stackoverflow thread: Embedding pybind11 with virtual environment

I set all the variables as @ipa did. Just look at the CMakeLists.txt and main.cpp:

cmake_minimum_required(VERSION 3.15)
project("cpp_google_trans")

set(
    PYTHON_EXECUTABLE "${CMAKE_HOME_DIRECTORY}/venv/Scripts/python.exe"
    CACHE FILEPATH "python virtual environment executable")
message(STATUS "PYTHON_EXECUTABLE is now: ${PYTHON_EXECUTABLE}")    #DEBUG

set(ENV{PYTHONPATH} "${CMAKE_HOME_DIRECTORY}/venv/Lib/site-packages")
message(STATUS "ENV{PYTHONPATH} is now: $ENV{PYTHONPATH}")  #DEBUG

set(
    ENV{PATH}
    "${CMAKE_HOME_DIRECTORY}/venv/Scripts/;${CMAKE_HOME_DIRECTORY}/venv/Lib"
)
message(STATUS "PATH is now: $ENV{PATH}")   #DEBUG

add_subdirectory(pybind11)

add_executable(app main.cpp)
target_link_libraries(app PRIVATE pybind11::embed)

main.cpp:

#include <iostream>
#include <string>
#include <pybind11/embed.h>
namespace py = pybind11;

int main() {
    py::scoped_interpreter guard{};

    auto sys = py::module::import("sys");
    py::print("Hello, World from Python!");
    py::print(sys.attr("executable"));
    py::print(sys.attr("version"));

    system("set PATH");

    std::cin.get();

    // system("powershell");
    // I was looking at the environment variables in the powershell, but there wasn't any variables I set in CMakeLists.txt

    // Here I get abort() when trying to import googletrans package.
    // The package is installed in the venv virtual environment.
    // When the package is installed globally there isn't any problem.
    py::object Translator = py::module::import("googletrans").attr("Translator");
    py::object translator = Translator();

    std::cout << "C++ program here!\nTell a phrase you want to translate to english: ";
    std::string phrase;
    std::getline(std::cin, phrase);

    std::cout << "\n\nThe phrase in english means: ";
    std::cout << translator.attr("translate")(py::cast(&phrase)).attr("text").cast<std::string>();
}

When looking at the cmake configuration output everything seems working well (instead of PythonLibs variable I don't know how to change):

-- Building for: Visual Studio 16 2019
-- The C compiler identification is MSVC 19.22.27812.2
-- The CXX compiler identification is MSVC 19.22.27812.2
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/VC/Tools/MSVC/14.22.27812/bin/Hostx64/x64/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/VC/Tools/MSVC/14.22.27812/bin/Hostx64/x64/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/VC/Tools/MSVC/14.22.27812/bin/Hostx64/x64/cl.exe
-- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Preview/VC/Tools/MSVC/14.22.27812/bin/Hostx64/x64/cl.exe -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- PYTHON_EXECUTABLE is now: D:/dev/Pybind/cpp_google_trans2/venv/Scripts/python.exe
-- ENV{PYTHONPATH} is now: D:/dev/Pybind/cpp_google_trans2/venv/Lib/site-packages
-- PATH is now: D:/dev/Pybind/cpp_google_trans2/venv/Scripts/;D:/dev/Pybind/cpp_google_trans2/venv/Lib    
-- Found PythonInterp: D:/dev/Pybind/cpp_google_trans2/venv/Scripts/python.exe (found version "3.7.3")    
-- Found PythonLibs: D:/Programs/VisualStudio/Shared/Python37_64/libs/Python37.lib
-- pybind11 v2.3.dev1
-- Configuring done
-- Generating done
-- Build files have been written to: D:/dev/Pybind/cpp_google_trans2/build

I also tried to set PYTHON_EXECUTABLE and PATH environment variables globally through control panel>system and security>system>advanced system settings>advanced>environment variables, but even this didn't help.

I am still pretty new to cmake, python and pybind11 and I am aware of that I don't know a lot of things. And maybe you've noticed this while reading this thread. ;P

Botje · Accepted Answer · 2019-07-05 20:12:04Z

5

The solution involves two parts.

Compile the virtualenv PYTHONPATH into your C++ program
In CMake, this can be done with target_compile_definitions. The custom path will be available as a preprocessor macro.

target_compile_definitions(app PRIVATE -DCUSTOM_SYS_PATH="\"${CMAKE_HOME_DIRECTORY}/venv/Lib/site-packages\"")

Update the Python sys.path right starting the interpreter.

This is pybind-specific, but ought to look something like:

py::module sys = py::module::import("sys");
sys.attr("path").attr("insert")(1, CUSTOM_SYS_PATH);

edited Jul 5, 2019 at 20:12

answered Jul 5, 2019 at 14:04

Botje

32.1k4 gold badges36 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Kacper O Over a year ago

I have a few questions about this solution: 1. What is unshift function (when I try to use it I get an error 'list' object has no attribute 'shift')? 2. Why didn't you just use append instead of unshift? 3. Why you use TEXT(CUSTOM_SYS_PATH) instead of just CUSTOM_SYS_PATH? TEXT(CUSTOM_SYS_PATH) is equal to "CUSTOM_SYS_PATH" though and as a path the string doesn't mean anything.

Botje Over a year ago

Derp. I was careless. Fixing... I replaced the unshift by insert at position 1, right after the . that pybind inserts. That ensures you pull in the version in the virtualenv as opposed to a system version. CUSTOM_SYS_PATH was meant to be a double-quoted string all along.

Collectives™ on Stack Overflow

Embedding python with pybind11. Virtual environment doesn't work

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related