I'm working on a Python project where I need to offload some computations to C++ for performance reasons. I am using Pybind11 to bridge between Python and C++. In my scenario, I am trying to run multiple C++ threads that call a callback function to modify a value in a Python object. However, I am encountering a deadlock and I'm not sure why. Here is a simplified version of my code:
Python Code:
import example
import threading
class MyClass:
def __init__(self, value=0):
self.value = value
self.lock = threading.Lock()
def python_callback(self, arg1: float, arg2: float) -> float:
with self.lock:
self.value += arg1
self.value += arg2
return float(self.value)
myclass = MyClass()
def cpp_thread():
example.call_cpp_thread(myclass, 1.0, 1.0, 10000)
def python_thread():
for i in range(40000):
myclass.python_callback(1, 1)
# Create and start threads
thread1 = threading.Thread(target=cpp_thread)
thread2 = threading.Thread(target=python_thread)
thread1.start()
thread2.start()
# Wait for threads to complete
thread1.join()
thread2.join()
print(myclass.value)
C++ Code:
#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <pybind11/stl.h>
#include <thread>
namespace py = pybind11;
void call_python_callback(py::object obj, float arg1, float arg2, int calltime) {
for (int i = 0; i < calltime; i++) {
obj.attr("python_callback")(arg1, arg2);
}
}
void call_cpp_thread(py::object obj, float arg1, float arg2, int calltime) {
std::thread t1(call_python_callback, obj, arg1, arg2, calltime);
std::thread t2(call_python_callback, obj, arg1, arg2, calltime);
std::thread t3(call_python_callback, obj, arg1, arg2, calltime);
std::thread t4(call_python_callback, obj, arg1, arg2, calltime);
t1.join();
t2.join();
t3.join();
t4.join();
}
PYBIND11_MODULE(example, m) {
m.def("call_python_callback", &call_python_callback, "Call a Python callback",
py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
m.def("call_cpp_thread", &call_cpp_thread, "Call cpp thread",
py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
}
Explanation: Python Class MyClass: It has a method python_callback that modifies the object's value. The method is protected with a threading.Lock to ensure thread safety of attribute value.
C++ Function call_python_callback: It repeatedly calls the Python callback function. The GIL is acquired automatically when call python code
C++ Function call_cpp_thread: It starts multiple threads that execute call_python_callback. Problem:
When I run the code, it results in a deadlock.
I have try to add an mutex in cpp to protect the call to python.
obj.attr("python_callback"), run that only once outside of the loop, unless you expect this to change. But that's a minor performance-related comment and it shouldn't be the cause of your problems.