1

I'm working on a Python project where I need to offload some computations to C++ for performance reasons. I am using Pybind11 to bridge between Python and C++. In my scenario, I am trying to run multiple C++ threads that call a callback function to modify a value in a Python object. However, I am encountering a deadlock and I'm not sure why. Here is a simplified version of my code:

Python Code:


import example
import threading

class MyClass:
    def __init__(self, value=0):
        self.value = value
        self.lock = threading.Lock()

    def python_callback(self, arg1: float, arg2: float) -> float:
        with self.lock:
            self.value += arg1
            self.value += arg2
        return float(self.value)

myclass = MyClass()

def cpp_thread():
    example.call_cpp_thread(myclass, 1.0, 1.0, 10000)

def python_thread():
    for i in range(40000):
        myclass.python_callback(1, 1)

# Create and start threads
thread1 = threading.Thread(target=cpp_thread)
thread2 = threading.Thread(target=python_thread)
thread1.start()
thread2.start()

# Wait for threads to complete
thread1.join()
thread2.join()

print(myclass.value)

C++ Code:

#include <pybind11/pybind11.h>
#include <pybind11/functional.h>
#include <pybind11/stl.h>
#include <thread>

namespace py = pybind11;

void call_python_callback(py::object obj, float arg1, float arg2, int calltime) {
    for (int i = 0; i < calltime; i++) {
        obj.attr("python_callback")(arg1, arg2);
    }
}

void call_cpp_thread(py::object obj, float arg1, float arg2, int calltime) {
    std::thread t1(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t2(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t3(call_python_callback, obj, arg1, arg2, calltime);
    std::thread t4(call_python_callback, obj, arg1, arg2, calltime);
    t1.join();
    t2.join();
    t3.join();
    t4.join();
}

PYBIND11_MODULE(example, m) {
    m.def("call_python_callback", &call_python_callback, "Call a Python callback",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
    m.def("call_cpp_thread", &call_cpp_thread, "Call cpp thread",
          py::arg("obj"), py::arg("arg1"), py::arg("arg2"), py::arg("calltime"));
}

Explanation: Python Class MyClass: It has a method python_callback that modifies the object's value. The method is protected with a threading.Lock to ensure thread safety of attribute value.

C++ Function call_python_callback: It repeatedly calls the Python callback function. The GIL is acquired automatically when call python code

C++ Function call_cpp_thread: It starts multiple threads that execute call_python_callback. Problem:

When I run the code, it results in a deadlock.

I have try to add an mutex in cpp to protect the call to python.

3
  • Why do you think it's a deadlock? What did you actually observe? Your interpretation may be flawed, hence the request for details. Commented Aug 1, 2024 at 16:09
  • BTW: obj.attr("python_callback"), run that only once outside of the loop, unless you expect this to change. But that's a minor performance-related comment and it shouldn't be the cause of your problems. Commented Aug 1, 2024 at 16:09
  • I don't know why it's deadlocking, but the code is trying to do something intensive, and I would recommend the following instead: Create a class in the C++ side (so that it holds state), and have the class do the operation you currently have on the python side (python_callback). Then make another function that is just a simple getter (to get "value") on the C++, and call the getter from the python side as often as you want to inspect what the current value is. That way, all the multithreaded manipulations are happening on the C++ side, and the python side is simply checking on the value. Commented Aug 1, 2024 at 16:16

1 Answer 1

0

The problem is like that When python call cpp code. It tries to create thread in cpp. The main thread of cpp holding the GIL and waiting for the sub-thread to finish (Because I call .join function on all thread).

In the sub-thread it need to GIL to execute the python call back.

This lead to an deadlock.

How to fix it: release the gil before join the thread.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.