Lazy iterators (generators) with asyncio

Question

I have a blocking, non-async code like this:

def f():
    def inner():
        while True:
            yield read()
    return inner()

With this code the caller can choose when to stop the function to generate data. How to change this to async? This solution doesn't work:

async def f():
    async def inner():
        while True:
            yield await coroutine_read()
    return inner()

... because yield can't be used in async def functions. If i remove the async from the inner() signature, I can't use await anymore.

You don't need yield read() when using asyncio, because await will do exactly this behind the scenes. This doesn't answer the question of course. — Sergei Lebedev
– Sergei Lebedev, Commented May 17, 2016 at 15:25
PEP-0492 doesn't cover coroutine-generators (which is what you want), so, since the PEP was only implemented in 3.5, I guess the answer is "there's no way to do this". — Sergei Lebedev
– Sergei Lebedev, Commented May 17, 2016 at 15:29
Attempt to implement way to yield inside async functions: stackoverflow.com/a/37572657/1113207 — Mikhail Gerasimov
– Mikhail Gerasimov, Commented Jun 1, 2016 at 15:21
Possible duplicate of How to use 'yield' inside async function? — rominf
– rominf, Commented Sep 18, 2016 at 11:57

Mikhail Gerasimov · Accepted Answer · 2019-04-07 08:52:45Z

10

Upd:

Starting with Python 3.6 we have asynchronous generators and able to use yield directly inside coroutines.

As noted above, you can't use yield inside async funcs. If you want to create coroutine-generator you have to do it manually, using __aiter__ and __anext__ magic methods:

import asyncio


# `coroutine_read()` generates some data:
i = 0
async def coroutine_read():
    global i
    i += 1
    await asyncio.sleep(i)
    return i


# `f()` is asynchronous iterator.
# Since we don't raise `StopAsyncIteration` 
# it works "like" `while True`, until we manually break.
class f:
    async def __aiter__(self):
        return self

    async def __anext__(self):
        return await coroutine_read()


# Use f() as asynchronous iterator with `async for`:
async def main():
    async for i in f():
        print(i)
        if i >= 3:
            break


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Output:

1
2
3
[Finished in 6.2s]

You may also like to see other post, where StopAsyncIteration uses.

edited Apr 7, 2019 at 8:52

answered May 17, 2016 at 21:32

Mikhail Gerasimov

40.1k20 gold badges126 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Paco Over a year ago

You should note that this approach is incorrect in Py 3.7 and later versions.

Collectives™ on Stack Overflow

Lazy iterators (generators) with asyncio

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related