Possible to run async function using exec?

Question

I am trying to call an async function from exec, I would expect it to work like:

def test():
    print("test")
    exec('def test2():\n    print("test")\ntest2()')
test()

Output:

test
test

So a function defined in exec is able to call itself, however we can't do such things in asyncio as:

async def test():
    print("test")
    exec('async def test2():\n    print("test")\nawait test2()')

We cant use await outside function as well as we cant call another loop from a running loop:

async def test():
    print("test")
    exec('async def test2():\n    print("test")\nasyncio.run(test2())')

Is there any solution to this problem?

Andrej Kesely · Accepted Answer · 2020-07-18 22:31:14Z

2

You can put current running loop as exec() global parameter:

import asyncio


async def test():
    print("test")
    loop = asyncio.get_running_loop()
    exec('async def test2():\n    print("test2")\nloop.create_task(test2())', {'loop': loop})

asyncio.run(test())

Prints:

test
test2

EDIT: To run asynchronously, you can return awaitable from the exec():

import asyncio

async def some_other_task():
    await asyncio.sleep(1)
    print('some_other_task')

async def test():
    loop = asyncio.get_running_loop()
    t = [None]

    exec('async def test2():\n    await asyncio.sleep(3);print("task in exec finished")\nt[0] = loop.create_task(test2())', {'asyncio': asyncio, 'loop': loop, 't': t})
    await asyncio.gather(some_other_task(), t[0])

asyncio.run(test())

Prints:

some_other_task           # <-- after 1 sec
task in exec finished     # <-- after 3 sec

edited Jul 18, 2020 at 22:31

answered Jul 18, 2020 at 22:13

Andrej Kesely

196k15 gold badges60 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

user2357112 Over a year ago

loop.create_task behaves differently from await, though. Particularly, it doesn't wait for the task to finish.

Strings Over a year ago

That solved problem i asked. But how would i run both of these tasks asynchronously ?

George Y Over a year ago

Hi Andrej! You seem to be the pro here. I have encountered a similar problem that a function calls itself, a coroutine restarts a coroutine on the end, which is quite easy in golang. This simple code does not work as expected, even by replacing create_task to ensure_future, could you solve this problem? onecompiler.com/python/3xxn4busd

Andrej Kesely Over a year ago

@GeorgeY It seems you're missing await keywords in asyncio.sleep and asyncio.create_task onecompiler.com/python/3xxncde7b

George Y Over a year ago

Interesting though, following your recommendation, the printing never ended.

|

user2357112 · Accepted Answer · 2020-07-18 22:29:30Z

2

Not possible. You would need some kind of "asynchronous exec" capable of suspending its own execution, and Python's syntax would need to support await outside an asynchronous function definition. Without that, there's no way for exec to return control to the event loop when it needs to suspend.

Andrej Kesely's answer does not work. It does not actually run the coroutine. It only arranges for the coroutine to run later.

answered Jul 18, 2020 at 22:29

user2357112

286k32 gold badges490 silver badges570 bronze badges

3 Comments

user2357112 Over a year ago

@AndrejKesely: It's closer to correct, but the coroutine still doesn't run until some point after exec finishes. The execution order means that, for example, the code inside the exec can't use the output of any coroutines it wants to run.

user4815162342 Over a year ago

"Not possible" might be a bit too strong. Couldn't an async_exec wrap the code into an async def x(): (adding indentation), execute the definition of x in a namespace and then return x() (or return await x() if async_exec is itself async)? Some details will be different, e.g. the variables defined at top-level won't be really global, but it might not matter for the OP's use case.

George Y Over a year ago

Hi! Could you tell me how to correct this code into allowing recursive coroutines called from the end of itself? I kinda know why the recursion stops at step 2, but is there a way in python asyncio that allows this usage? onecompiler.com/python/3xxn4busd

John Lehmann · Accepted Answer · 2022-01-08 05:09:21Z

I needed something like this for a project, and wrote an async version of code.InteractiveConsole. I also wanted to capture output, so used an idea from twisted.conch.Manhole

This is an awful hack. It only works for async code lines that start with "await". I haven't worked out how to handle the form x = await func().

import asyncio

def handle(output):
    print(f"** {output}", end="")

async def nested():
    return 42

async def main():
    localz = {"nested": nested}
    cons = AsyncConsole(handle, localz)
    await cons.interact("a = 10")
    await cons.interact("b = 20")
    await cons.interact("def fun(a, b):")
    await cons.interact("    return a + b")
    await cons.interact("")
    await cons.interact("fun(a, b)")
    await cons.interact("await nested()")

    del localz['__builtins__']
    print(f"l: {localz}")

asyncio.run(main())

Output:

** >>> a = 10
** >>> b = 20
** >>> def fun(a, b):
** ...     return a + b
** ...
** >>> fun(a, b)
30
** >>> await nested()
42
l: {'nested': <function nested at 0x100ab0820>, 'a': 10, 'b': 20, 'fun': <function fun at 0x101059480>, '_': 42}

AsyncConsole:

import string
import code
import sys
import io


class AsyncConsole(code.InteractiveConsole):
    def __init__(self, handler, locals: dict = None, filename="<console>"):
        super().__init__(locals, filename)
        self.handler = handler
        self.filename = filename
        self.output = io.StringIO()
        self.prompt1 = ">>> "
        self.prompt2 = "... "
        self.prompt = self.prompt1
        self.is_async = False

    async def runcode(self, code):
        orighook, sys.displayhook = sys.displayhook, self.displayhook
        try:
            origout, sys.stdout = sys.stdout, self.output
            try:
                exec(code, self.locals)
                if self.is_async:
                    coro = self.locals["_"]
                    obj = await coro
                    self.locals["_"] = obj
                    if obj is not None:
                        self.write(repr(obj))
            except SystemExit:
                raise
            except Exception:
                self.showtraceback()
            finally:
                sys.stdout = origout
        finally:
            sys.displayhook = orighook

    def displayhook(self, obj):
        self.locals["_"] = obj
        if obj is not None and not self.is_async:
            self.write(repr(obj))

    def write(self, data):
        self.output.write(data)

    async def runsource(self, source, filename="<input>", symbol="single"):
        try:
            code = self.compile(source, filename, symbol)
        except (OverflowError, SyntaxError, ValueError):
            # Case 1
            self.showsyntaxerror(filename)
            return False

        if code is None:
            # Case 2
            return True

        # Case 3
        await self.runcode(code)
        return False

    async def push(self, line):
        self.buffer.append(line)
        source = "\n".join(self.buffer)
        more = await self.runsource(source, self.filename)
        if not more:
            self.resetbuffer()
        return more

    async def interact(self, line):
        self.is_async = line.startswith("await ")
        self.output = io.StringIO()
        self.output.write(f"{self.prompt}{line}\n")
        if self.is_async:
            line = line[6:]
        r = await self.push(line)
        self.prompt = self.prompt2 if r else self.prompt1
        if not r and "_" in self.locals and self.locals["_"]:
            self.output.write("\n")
        self.handler(self.output.getvalue())
        return self.prompt

Collectives™ on Stack Overflow

Possible to run async function using exec?

3 Answers 3

6 Comments

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

6 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related