Raising python exception from native thread

Question

I have a C++ class Runner that encapsulates running a native thread. This class is exposed to Python via pybind11, so start_thread is called from Python.

class Runner {
public:
    void start_thread() {
        thread_ = std::jthread([]{
            try {
                while (true) {
                    // do something
                }
            } catch (...) {
                pybind11::gil_scoped_acquire gil;
                pybind11::set_error(PyExc_RuntimeError, "Something horrible happend");
            }
        });
    }

private:
    std::jthread thread_;
};

Exception might be thrown in native thread, is there a way to reraise an exception from a native thread to Python?

I was thinking about using pybind11::set_error, but is it the right way? And if so what are the guarantees on when Python realises that there is an exception? If exceptions are checked as soon as Gil is released by a native thread - then this solution should be fine.

What are you hoping will happen with this exception? Do you want to catch it somewhere? Trying to handle an exception from one thread in another thread would already take a lot of manual work even if both threads were Python threads. — user2357112
– user2357112, Commented Oct 9, 2024 at 9:58
Alternative to exception would be a std::terminate, I'd hope that reraising exception in Python will allow Python interpreter to gracefully shutdown, calling potential finally blocks. Or if it's non terminal exception, Python could recover from that and restart native thread. — DoctorMoisha
– DoctorMoisha, Commented Oct 9, 2024 at 10:08
A more "c++ native" way is to throw the exception in c++ and register a pybind11 custom exception translator so pybind11 will translate it to a python exception — Ahmed AEK
– Ahmed AEK, Commented Oct 9, 2024 at 10:11
Note pybind11 has a net that translates all "unknown" exceptions to RuntimeError, you don't need to add one. — Ahmed AEK
– Ahmed AEK, Commented Oct 9, 2024 at 10:18
@AhmedAEK exception can't be raised from a native thread without violating GIL. More precisely - throwing exception in a native thread causes std::terminate because it's handled by C++ runtime, rather than Python — DoctorMoisha
– DoctorMoisha, Commented Oct 9, 2024 at 10:21

pptaszni · Accepted Answer · 2024-10-09 10:44:14Z

Like you noticed, you cannot let C++ exception escape jthread, because it will be handled on C++ side and cause crash. Relatively good way would be to store std::exception_ptr inside your Runner and provide additional interface to rethrow it in the caller side:

class Runner {
public:
  void start_thread() {
    thread_ = std::jthread([this] {
      try {
        throwSomething();
      } catch (const std::exception &e) {
        ePtr_ = std::current_exception();
      }
    });
  }
  void rethrow_if_needed() const {
    if (ePtr_) {
      std::rethrow_exception(ePtr_);
    }
  }

private:
  void throwSomething() { throw std::runtime_error("crit failure"); }
  std::jthread thread_;
  std::exception_ptr ePtr_;
};

Then your python code could look like:

runner = Runner()
runner.start_thread()

try:
    time.sleep(1)
    runner.rethrow_if_needed()
except Exception as e:
    print("caught: ", e)

std::runtime_error should be automatically translated to pythonic RuntimeError.

Collectives™ on Stack Overflow

Raising python exception from native thread

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related