Python - Run multiple async functions simultaneously

Question

I'm essentially making a pinger, that makes has a 2d list, of key / webhook pairs, and after pinging a key, send the response to a webhook

the 2d list goes as follows:

some_list = [["key1", "webhook1"], ["key2", "webhook2"]]

My program is essentially a loop, and I'm not too sure how I can rotate the some_list data, in the function.

Here's a little demo of what my script looks like:

async def do_ping(some_pair):
    async with aiohttp.ClientSession() as s:
        tasks = await gen_tasks(s, some_pair)
        results = await asyncio.gather(*tasks*)
        sleep(10)
        await do_ping(some_pair)

I've tried:

async def main(): 
    for entry in some_list: 
        asyncio.run(do_ping(entry))

but due to the do_ping function being a self-calling loop, it just calls the first one over and over again, and never gets to the ones after it. Hoping to find a solution to this, whether it's threading or alike, and if you have a better way of structuring some_list values (which I assume would be a dictionary), feel free to drop that feedback as well

Angel · Accepted Answer · 2024-10-08 07:06:21Z

8

You made your method recursive await do_ping(some_pair), it never ends for the loop in main to continue. I would restructure the application like this:

async def do_ping(some_pair):
    async with aiohttp.ClientSession() as s:
        while True:
            tasks = await gen_tasks(s, some_pair)
            results = await asyncio.gather(*tasks)
            await asyncio.sleep(10)


async def main(): 
    tasks = [do_ping(entry) for entry in some_list]
    await asyncio.gather(*tasks)


if __name__ == "__main__":
    asyncio.run(main())

Alternatively you could move the repeat and sleeping logic into the main:

async def do_ping(some_pair):
    async with aiohttp.ClientSession() as s:
        tasks = await gen_tasks(s, some_pair)
        results = await asyncio.gather(*tasks)


async def main(): 
    while True:
        tasks = [do_ping(entry) for entry in some_list]
        await asyncio.gather(*tasks)
        await asyncio.sleep(10)


if __name__ == "__main__":
    asyncio.run(main())

You could also start the tasks before doing a call to sleep, and gather them afterwards. That would make the pings more consistently start at 10 second intervals instead of being 10 seconds + the time it takes to gather the results:

async def main(): 
    while True:
        tasks = [
            asyncio.create_task(do_ping(entry))
            for entry in some_list
        ]
        await asyncio.sleep(10)
        await asyncio.wait(tasks)

EDIT As pointed out by creolo you should only create a single ClientSession object. See https://docs.aiohttp.org/en/stable/client_reference.html

Session encapsulates a connection pool (connector instance) and supports keepalives by default. Unless you are connecting to a large, unknown number of different servers over the lifetime of your application, it is suggested you use a single session for the lifetime of your application to benefit from connection pooling.

async def do_ping(session, some_pair):
    tasks = await gen_tasks(session, some_pair)
    results = await asyncio.gather(*tasks)

async def main(): 
    async with aiohttp.ClientSession() as session:
        while True:
            tasks = [
                asyncio.create_task(do_ping(session, entry))
                for entry in some_list
            ]
            await asyncio.sleep(10)
            await asyncio.wait(tasks)

edited Oct 8, 2024 at 7:06

Angel

3,1724 gold badges27 silver badges45 bronze badges

answered Apr 1, 2022 at 17:53

flakes

24.1k8 gold badges52 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

neiii Over a year ago

But wouldn't that cause for the pinger to run only once per pair?

flakes Over a year ago

@neiii Notice that I added in a while True. Each do_ping task will run forever unless there is an exception or signal interrupt.

neiii Over a year ago

this actually works, thanks so much!

flakes Over a year ago

@neiii glad to help! Also do note, when using asyncio its very important to do await asyncio.sleep(10) rather than time.sleep(10). The latter will block the interpreter and will block any other tasks from doing work.

creolo Over a year ago

It is also recommended to use a single ClientSession() over the lifetime of the application as stated here: docs.aiohttp.org/en/stable/client_reference.html

Collectives™ on Stack Overflow

Python - Run multiple async functions simultaneously

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related