Python asyncio - How to create task list and use it in the event loop?

Question

I'm not very experienced in Python asyncio, although synchronous Python is going well. I have a function, which creates a task list, and another function which is to be called with tasks in this list:

import asyncio

async def createTasks(some_dict):
    coroutines = []
    # some_dict can have an arbitrary number of items
    for item in some_dict:
        coroutines.append(executeOneTask(item))
    tasks = await asyncio.gather(*coroutines, return_exceptions=True)
    return tasks

async def executeOneTask(item):
    # execute a task asynchronously
    return

Here's the part where you are free to correct me if I'm wrong.

Now, my understanding of asyncio is that I need an event loop to execute an asynchronous function, which means that to asyncio.gather I need to await it that means this needs to happen inside an async function. OK, so I need an event loop to create my list of asynchronous tasks that I actually want to execute asynchronously.

If my understanding of event loops is correct, I cannot easily add tasks inside an event loop to that same event loop. Let's assume that I have an asynchronous main() function which is supposed to first retrieve a list of asynchronous tasks using createTasks() and then create an amount (equal to list length) of asynchronous tasks to be run by utilizing executeOneTask().

How should I approach the construction of such a main() function? Do I need multiple event loops? Can I create a task list some other way, which enables easier code?

Side note: The approach I have set up here might be a very difficult or upside-down way to solve the problem. What I aim to do is to create a list of asynchronous tasks and then run those tasks asynchronously. Feel free to not follow the code structure above if a smart solution requires that.

Thanks!

Asyncio handles the event loop. All you need to call is asyncio.run(createTasks(some_dict) — balki
– balki, Commented Apr 2, 2020 at 10:14
Using this method, how should the executeOneTask() be called? Or are the tasks executed bu just gathering them inside asyncio.run()? — Aarni Joensuu
– Aarni Joensuu, Commented Apr 2, 2020 at 10:29
Calling an async function returns a coroutine object and does not actually run the function, so your code is correct. gather effectively awaits all its arguments and thus each instance of executeOneTask is executed in there. — Tronic
– Tronic, Commented Apr 2, 2020 at 11:00

Tronic · Accepted Answer · 2020-04-02 10:17:06Z

1

You should only use one event loop in the entire application. Start the main function by asyncio.run(main()) and asyncio creates a loop for you. With Python 3.8 you rarely need to access or use the loop directly but with older versions you may obtain it by asyncio.get_event_loop() if using loop methods or some functions that require loop argument.

Do note that IPython (used in Spyder and Jupyter) also runs its own loop, so in those you can directly call and await without calling asyncio.run.

If you only wish to do async programming but don't specifically need to work with asyncio, I would recommend checking out https://trio.readthedocs.io/ which basically does the same things but is much, much easier to use (correctly).

answered Apr 2, 2020 at 10:17

Tronic

1,42615 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python asyncio - How to create task list and use it in the event loop?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related