How to make request without blocking (using asyncio)?

Question

I would like to achieve the following using asyncio:

# Each iteration of this loop MUST last only 1 second
while True:
    # Make an async request

    sleep(1)

However, the only examples I've seen use some variation of

async def my_func():
    loop = asyncio.get_event_loop()
    await loop.run_in_executor(None, requests.get, 'http://www.google.com')

loop = asyncio.get_event_loop()
loop.run_until_complete(my_func())

But run_until_complete is blocking! Using run_until_complete in each iteration of my while loop would cause the loop to block.

I've spent the last couple of hours trying to figure out how to correctly run a non-blocking task (defined with async def) without success. I must be missing something obvious, because something as simple as this should surely be simple. How can I achieve what I have described?

The loop itself will always be blocking. Ideally you’d want to put your while inside a coroutine run by your loop, but without a better idea of how your code is supposed to work, it’s hard to say much more. — dirn
– dirn, Commented Oct 11, 2017 at 17:14
hello, I find myself in this exact situation where All I want is to have normal synchronous execution... (inside a while loop) ...with the ability to call async methods when needed. I don't care about the result. Were you able to achieve this??? please, thanks — Scaramouche
– Scaramouche, Commented Mar 17, 2020 at 20:05
In JavaScript this is trivial to achieve, pity Python makes it so hard. — basickarl
– basickarl, Commented May 14, 2021 at 11:21

user1527491 · Accepted Answer · 2017-10-11 21:01:58Z

3

run_until_complete runs the main event loop. It's not "blocking" so to speak, it just runs the event loop until the coroutine you passed as a parameter returns. It has to hang because otherwise, the program would either stop or be blocked by the next instructions.

It's pretty hard to tell what you are willing to achieve, but this piece code actually does something:

async def my_func():
    loop = asyncio.get_event_loop()
    while True:
        res = await loop.run_in_executor(None, requests.get, 'http://www.google.com')
        print(res)
        await asyncio.sleep(1)
loop = asyncio.get_event_loop()
loop.run_until_complete(my_func())

It will perform a GET request on Google homepage every seconds, popping a new thread to perform each request. You can convince yourself that it's actually non-blocking by running multiple requests virtually in parallel:

async def entrypoint():
    await asyncio.wait([
        get('https://www.google.com'),
        get('https://www.stackoverflow.com'),
    ])

async def get(url):
    loop = asyncio.get_event_loop()
    while True:
        res = await loop.run_in_executor(None, requests.get, url)
        print(url, res)
        await asyncio.sleep(1)

loop = asyncio.get_event_loop()
loop.run_until_complete(entrypoint())

Another thing to notice is that you're running requests in separate threads each time. It works, but it's sort of a hack. You should rather be using a real asynchronus HTTP client such as aiohttp.

edited Oct 11, 2017 at 21:01

answered Oct 11, 2017 at 20:05

user1527491

96511 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dotl Over a year ago

Thanks, this makes things a little clearer, but wouldn't this also make my while loop be asynchronous (when I don't want it to be, as shown in my example)? All I want is to have normal synchronous execution with the ability to call async methods when needed. I don't care about the result. With threading this was simple - Thread().start(). Perhaps using asyncio just needs a major mindset change on my part.

user1527491 Over a year ago

asyncio indeed requires you to change the whole paradigm of your application. Imagine you call run_until_complete and it does not hang. Just after it, you might call time.sleep(5) which is a blocking function. In fact, any function you would call after would never give a chance to the event loop to poll. Coroutines would never execute because the event loop isn't running/polling (remember: all this happens on one only thread). If you need to call blocking functions later on, do it in another thread (using run_in_executor) or run the whole event loop in another thread.

Eric Aya · Accepted Answer · 2022-04-21 20:47:50Z

1

This is Python 3.10

asyncio is single threaded execution, using await to yield the cpu to other function until what is await'ed is done.

    import asyncio
    async def my_func(t):
        print("Start my_func")
        await asyncio.sleep(t)  # The await yields cpu, while we wait
        print("Exit my_func")

    async def main():
        asyncio.ensure_future(my_func(10)) # Schedules on event loop, we might want to save the returned future to later check for completion.
        print("Start main")
        await asyncio.sleep(1) # The await yields cpu, giving my_func chance to start.
        print("running other stuff")
        await asyncio.sleep(15)
        print("Exit main")

    if __name__ == "__main__":
        asyncio.run(main())  # Starts event loop

edited Apr 21, 2022 at 20:47

Eric Aya

70.2k36 gold badges190 silver badges266 bronze badges

answered Apr 21, 2022 at 20:22

Pieter

2,19219 silver badges20 bronze badges

Collectives™ on Stack Overflow

How to make request without blocking (using asyncio)?

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related