How to integrate asyncronous python code into synchronous function?

Question

I have an external library that uses requests module to perform http requests. I need to use the library asynchronously without using many threads (it would be the last choice if nothing else works). And I can't change its source code either. It would be easy to monkey-patch the library since all the interacting with requests module are done from a single function, but I don't know if I can monkey-patch synchronous function with asynchronous one (I mean async keyword).

Roughly, the problem simplifies to the following code:

import asyncio
import aiohttp
import types
import requests


# Can't modify Library class.
class Library:
    def do(self):
        self._request('example.com')

        # Some other code here..

    def _request(self, url):
        return requests.get(url).text


# Monkey-patched to this method.
async def new_request(self, url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as response:
            return await response.text()


async def main():
    library = Library()

    # Do monkey-patch.
    library._request = types.MethodType(new_request, library)

    # Call library asynchronously in hope that it will perform requests using aiohttp.
    asyncio.gather(
        library.do(),
        library.do(),
        library.do()
    )

    print('Done.')

asyncio.run(main())

But as expected, it doesn't work. I get TypeError: An asyncio.Future, a coroutine or an awaitable is required on asyncio.gather call. And also RuntimeWarning: coroutine 'new_request' was never awaited on self._request('example.com').

So the question is: is it possible to make that code work without modifying the Library class' source code? Otherwise, what options do I have to make asynchronous requests using the library?

Artyom Vancyan · Accepted Answer · 2022-10-14 15:14:15Z

1

Is it possible to make that code work without modifying the Library class' source code? Otherwise, what options do I have to make asynchronous requests using the library?

Yes, it is possible, and you even do not need monkey-patching to perform that. You should use asyncio.to_thread to make the synchronous do method of Library an asynchronous function (coroutine). So the main coroutine should look like this:

async def main():
    library = Library()

    await asyncio.gather(
        asyncio.to_thread(library.do),
        asyncio.to_thread(library.do),
        asyncio.to_thread(library.do)
    )

    print('Done.')

Here the asyncio.to_thread wraps the library.do method and returns a coroutine object avoiding the first error, but you also need await before asyncio.gather.

NOTE: If you are going to check my answer with the above example, please do not forget to set a valid URL instead of 'example.com'.

Edit

If you do not want to use threads at all, I would recommend an async wrapper like the to_async function below and replace asyncio.to_thread with that.

async def to_async(func):
    return func()


async def main():
    library = Library()

    await asyncio.gather(
        to_async(library.do),
        to_async(library.do),
        to_async(library.do),
    )

edited Oct 14, 2022 at 15:14

answered Oct 14, 2022 at 12:59

Artyom Vancyan

5,4023 gold badges17 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

g00dds Over a year ago

That's not exactly what I wanted, because to_thread implies that it uses multithreading (uses ThreadPoolExecutor internally as default executor) instead of asynchronous asyncio loop. to_thread just lets you to await the result from a thread asynchronously. But it still executes in different threads. That's what I wanted to avoid if it's possible. But anyway, thank you for your answer. I didn't know about to_thread function and since my entire application uses asyncio I will use to_thread if I don't find a better solution.

Artyom Vancyan Over a year ago

@g00dds, I got your point and updated the answer. Please check the Edit section of my answer.

g00dds Over a year ago

In this case the function seems never to be called at all. The whole source code file I ran to test this: pastebin.com/5pv3e59a. I even put a print in the wrapper function, it never gets called either.

Artyom Vancyan Over a year ago

Oh, sorry, it was initially implemented as a decorator, then I tried to use it as an ordinary function wrapper. Please check once again. This one is more simple.

g00dds Over a year ago

I applied the changes. When monkey-patch is used, instead of the response _request function returns coroutine object so the further code fails with errors related to invalid response. Without the monkey-patch (just original function) code works, but synchronously, as if nothing was changed. The code(monkey-patching commented): pastebin.com/UveEi4Ws

|

Artyom Vancyan · Accepted Answer · 2022-10-14 16:59:41Z

0

I thought to release another answer as it solves the problem in another way.

So why do not extend the default behavior of the Library class and make the do and _request methods polymorph?

class AsyncLibrary(Library):

    async def do(self):
        return await self._request('https://google.com/')

    async def _request(self, url):
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as response:
                return await response.text()


async def main():
    library = AsyncLibrary()

    await asyncio.gather(
        library.do(),
        library.do(),
        library.do(),
    )

answered Oct 14, 2022 at 16:59

Artyom Vancyan

5,4023 gold badges17 silver badges38 bronze badges

2 Comments

g00dds Over a year ago

This is just like monkey-patching, but is done in another way. The other library's methods use overrided methods as synchronous. They do not await for the result, but instead use the returned value, which is coroutine for all async functions. This also means that I need to override all methods called until the _request call (do is not the first called method in the real library). The second problem is that the methods that use _request method do not consist of a single call to _request, but contains a lot of other logic and it would be hard to reimplement those methods.

Artyom Vancyan Over a year ago

I understand that the do is not the first called method in the real library and by changing its behavior can change others' behaviors too. I do not know what exact library you are using and my solution is based on your example. If you cannot or do not want change the original implementation of the do method, then you have to keep the _request method as a synchronous method because it is used in that way. In case you evaluate the new_request from overridden _request it will not make requests be sent asynchronous, and here the solution with to_thread became the best one. Thanks :)

jwal · Accepted Answer · 2022-10-14 21:36:20Z

No way to do this with requests without threads but you can limit the number of threads active at any one time to address your without using many threads requirement.

import asyncio
import requests

# Can't modify Library class.
class Library:
    def do(self):
        self._request('http://example.com')
    def _request(self, url):
        return requests.get(url).text

async def as_thread(semaphore, func):
    async with semaphore:  # limit the number of threads active
        await asyncio.to_thread(func)

async def main():
    library = Library()
    semaphore = asyncio.Semaphore(2)  # limit to 2 for example
    tasks = [library.do] * 10  # pretend there are a lot of sites to read
    await asyncio.gather(
        *[as_thread(semaphore, x) for x in tasks]
    )
    print('Done.')

asyncio.run(main())

Collectives™ on Stack Overflow

How to integrate asyncronous python code into synchronous function?

3 Answers 3

Edit

7 Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Edit

7 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related