A common pattern with asyncio, like the one shown here, is to add a collection of coroutines to a list, and then asyncio.gather them.
For instance:
async def some_task(i):
# Do something asynchronously with i
tasks = [some_task(i) for i in range(100)]
loop.run_until_complete(asyncio.gather(**tasks))
Here, the execution order of this code is such that none of the tasks are running while we build up the list. We add task 1 to the list, then task 2, etc. and then we add the tasks 1-100 to the event loop.
However, I want task creation itself to be part of the event loop. I want task 1 to be scheduled immediately as it's created, and then when task is waiting for something on another thread, return to task creation and create task 2 and add it to the event loop.
I believe this would give me better concurrency from my async code. Is this possible?
For example, my first thought would be to put task creation into a coroutine and schedule tasks as they are created:
async def some_task(i):
# Do something asynchronously with i
async def generate_tasks(loop):
tasks = []
for i in range(100):
task = loop.create_task(some_task(i))
tasks.append(loop)
await asyncio.gather(**tasks)
loop.run_until_complete(generate_tasks())
However, because my generate_tasks never uses await, execution is never passed back to the event loop, so the entirety of generate_tasks will run before some_task() is run at all.
But then, if I await each task as they are created, it will wait for each task to complete before moving on to the next task, giving me no concurrency at all!
async def generate_tasks(loop):
tasks = []
for i in range(100):
await some_task(i)
loop.run_until_complete(generate_tasks())
tasks = [some_task(i) for i in range(100)], all 100 tasks have been created.some_task(i)returns a task object. And whatasyncio.gatherdoes is adding all tasks into loop and awaiting all of them. So you won't get better concurrency.