How to call Rust async method from Python?

Question

I want to use a Rust async method in Python. I'm trying to use PyO3 or rust-cpython.

For example, for sync Rust functions, I can use,

#[pyfunction]
fn myfunc(a: String) -> PyResult<String> {
   let mut contents = String::new();
   contents = a.to_string() + " appended";
   Ok((contents))
}

#[pymodule]
fn MyModule(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(urlshot))?;
    Ok(())
}

For async methods, how can I do it? For example, I want to call the following method in Python,

async fn hello_world() {
    println!("hello, world!");
}

To be able to await an async function, you must be inside an async function (defined with async def in Python). Who is calling that function in the use case you envision? You won't be able to mix&match Python asyncio and Rust tokio (or others), but if your Python async function is itself awaited from async Rust, awaiting a Rust async fn should at least be possible. — user4815162342
– user4815162342, Commented Jun 28, 2020 at 16:45

mkrieger1 · Accepted Answer · 2023-11-23 18:36:41Z

9

Since there was no easy way of solving this issue (at least, I hadn't found), I converted my async method to sync one. And called it on Python side as,

async fn my_method(s: &str) -> Result<String, Error> {
    // do something
}

#[pyfunction]
fn my_sync_method(s: String) -> PyResult<String> {
    let mut rt = tokio::runtime::Runtime::new().unwrap();
    let mut contents = String::new();
    rt.block_on(async {
        result = format!("{}", my_sync_method(&s).await.unwrap()).to_string();
    });
   Ok((result))
}

#[pymodule]
fn MyModule(py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pyfunction!(my_sync_method))?;
    Ok(())
}

In the Cargo.toml file, I added the following dependencies,


[dependencies.pyo3]
git = "https://github.com/PyO3/pyo3"
features = ["extension-module"]

After running cargo build --release, target/release/libMyModule.so binary file is generated. Rename it as MyModule.so and it now can be imported from Python.

import MyModule
result = MyModule.my_sync_method("hello")

Using setuptools-rust, I could bundle it as an ordinary Python package.

All of the above code and commands are tested on newly-released Linux Mint 20. On MacOS, the binary file will be libMyModule.dylib.

edited Nov 23, 2023 at 18:36

mkrieger1

24.2k7 gold badges68 silver badges84 bronze badges

answered Jun 28, 2020 at 13:54

DumTux

7341 gold badge11 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ruohola Over a year ago

Would help others if you would post your Python code also which calls these functions!

DumTux Over a year ago

@ruohola I added some more instructions. If you wanted a fully working sample, let me know.

ruohola Over a year ago

Nice stuff, ty!

Kraigolas Over a year ago

To future readers: if you take the dependencies above, you should write branch = "main" under the link to the pyo3 repository. Looks like they've changed the branch name, and rust still defaults to assuming the branch is master.

whilrun · Accepted Answer · 2020-06-28 10:35:05Z

4

If you want to use Python to control Rust's async function, I don't think it will work (Or at least it is very complicated, as you need to connect two different future mechanism). For async functions, Rust compiler will maintain a state machine to manage the coroutines run correctly under await's control. This is an internal state of Rust applications and Python cannot touch it. Similarly Python interpreter also has a state machine that cannot be touched by Rust.

I do found this topic about how to export an async function using FFI. The main idea is to wrap the async in a BoxFuture and let C control the timing of returning it to Rust. However, you cannot use BoxFuture in PyO3 since its pyfunction macro cannot convert a function returns BoxFuture to a Python callback. You may try to create a library using FFI and use python's cffi module to load it.

edited Jun 28, 2020 at 10:35

answered Jun 28, 2020 at 10:20

whilrun

1,74415 silver badges24 bronze badges

3 Comments

DumTux Over a year ago

Thank you. I also found Python Async/Await Interface in PyO3 documentation. Could you explain what this class is for?

whilrun Over a year ago

@HotteShen This interface is to coopearate with Python's state machine. If you read the Python's document you will find that these functions need to return an awaitable object to Python. You know that Python uses duck type, so as long as you can satisfy these structure's requirement Python can use your extension function with await. But it does not mean you can just return a Rust's async function, since their implementations of future are different.

DumTux Over a year ago

Understood. Thank you!

targetXING · Accepted Answer · 2023-11-23 18:29:22Z

I found this tutorial in the pyo3 documentation: https://pyo3.rs/v0.20.0/ecosystem/async-await

If you need to call a Rust async function that returns a result (String, None, etc) from python, it seems you need to access Python's GIL:

lib.rs

use pyo3::prelude::*;

async fn rust_sleep() {
    async_std::task::sleep(std::time::Duration::from_secs(1)).await;
}

#[pyfunction]
fn call_rust_sleep(py: Python<'_>) -> PyResult<&PyAny> {
    pyo3_asyncio::async_std::future_into_py(py, async move {
        rust_sleep().await;
        Ok(Python::with_gil(|py| py.None()))
    })

example.py

from example import call_rust_sleep

async def rust_sleep():
    await call_rust_sleep()

To return back a string from the async function:

Ok(result) => Ok(Python::with_gil(|py| "hello world".to_string()))

Collectives™ on Stack Overflow

How to call Rust async method from Python?

3 Answers 3

4 Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related