python: iterate over either a list or an async generator

Question

Since iterators were introduced in python, it's always been possible to not care whether you are dealing with an iterator or a list:

from random import random

def gen_list():
    print('gen')
    for i in range(10):
        yield i

def return_list():
    print('return')
    return [i for i in range(10)]


if random() > 0.5:
    x = gen_list()
else:
    x = return_list()

for i in x:
    pass

PEP 492 introduced asynchronous iterators and the async for syntax. What I can't see is any justification for the new burden of adding syntax to the consumer of the async iterator.

In my code, I sometimes am dealing with a list (from a cache), and sometimes with an async generator:

import asyncio
from random import random

def is_small_and_in_cache():
    if random() > 0.5:
        print('in fake cache')
        return [i for i in range(10)]

async def get_progressively():
    print('gen')
    for i in range(10):
        # e.g. an await here
        await asyncio.sleep(0.1)
        yield i

async def main():
    x = is_small_and_in_cache()
    if x is None:
        x = get_progressively()

    async for i in x:
        pass

asyncio.run(main())

But the above fails (half the time) with TypeError: 'async for' requires an object with __aiter__ method, got list.

Main Question: How to write this so that we can deal with either? Should I try to convert the list to a dummy async generator, or wrap the async generator so that it produces a list?

Side Quest: Are there any proposals to get rid of the (clearly unpythonic, to me) async for construct, i.e. why can't a regular for loop handle an asynchronous generator? Has Python3x lost it's way in terms of usability??

Davis Herring · Accepted Answer · 2021-06-11 04:19:39Z

9

The syntax exists to warn you that your “loop” might actually include suspending your entire call, allowing other code to run, so that you know to have appropriate data in a consistent state at the top of each iteration. It’s not going anywhere.

Of course, a coroutine doesn’t have to suspend, and you can use that to make wrapping any iterable trivial:

async def desync(it):
  for x in it: yield x

This is more generally useful than the opposite number which—still asynchronously, as it must—gathers into a list:

async def gather(ai):
  ret=[]
  async for x in ai: ret.append(x)
  return ret

since it allows for proper interleaving in the fully asynchronous case.

answered Jun 11, 2021 at 4:19

Davis Herring

42k4 gold badges59 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

EoghanM Over a year ago

hmmm, although it's academic, for i in await x: would better get across this idea, as it's not the loop that relinquishes control, but the generator. This would also better mirror the caller=>await/callee=>async of the existing async function system.

Davis Herring Over a year ago

@EoghanM: for i in await x: already means something, though: that x is some awaitable (e.g., a suspended asynchronous generator) that provides a whole (synchronous) iterable in one go.

Collectives™ on Stack Overflow

python: iterate over either a list or an async generator

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related