Python: Separating asyncio coroutine declaration from event looping/execution

Question

Python version: 3.5.2

I'm trying to implement an easy to use interface using asyncIO for my end users.

In my model, the end user defines an async coroutine that operates on singular object(s).

e.g., user_func(addr_object, property_object)

Now, I would like to do something like this...

User Code:

calling this file "user.py"

# instantiating my_api object
batch_runner = my_api.BatchRunner()

# coro_list can have more than one user defined function
coro_list = [user_func(addr_object, property_object)]

# this method takes care of creating the eventloop and running it
batch_runnner.run(coro_list)


# all user defined coroutines live in this file "user.py"
async def user_func(addr_object, property_object):
    ``` operate on the objects
    # await on some operation

P.S.: addr_object and property_object can be assigned to None in user.py to keep Python happy

My API code:

# calling this file "my_api.py"

class BatchRunner(object):
...
    def run(self, coro_list):
        new_coro_list = list()
        # user can define multiple coroutines, but same type and length of input arguments on all of them
        for coros in coro_list:
            # Say I've 10 different addr_objects for example
            for addr_obj in addr_obj_list:
                # and say another 5 property_objects
                for prop_obj in prop_obj_list:
                    # how to modify coro_list to work on each of these addr_obj and prop_obj?
                    # while maintaining reference to the user defined coroutine in user.py?
                    new_coro_list.append(coros(addr_obj, prop_obj)

        eventloop = asyncio.get_event_loop()
        eventloop.run_until_complete(new_coro_list)

My questions are:

Is it possible to maintain reference to a coroutine object declared in another file (user.py) and modify the value of its input arguments before actual eventloop execution inside of my_api.py? i.e., same function call, but with modified argument values.
If that is not possible, is there a better way to hide the iteration logic under the hood? (Could be using **kwargs with user function name being passed as a string)

Udi · Accepted Answer · 2017-02-12 10:52:39Z

1

try this:

def get_args():
    for i in range(10):
        for c in "ABCD":
            yield i, c

def batch_runner(funcs):
    tasks = [func(*args) for func in funcs for args in get_args()]
    loop = asyncio.get_event_loop()
    return loop.run_until_complete(asyncio.gather(*tasks))

usage:

async def user_func1(i, c):
    print(">U1", i, c)
    await asyncio.sleep(i / 10)
    print("<U1", i, c)
    return("U1", i, c)    

async def user_func2(i, c):
    print(">U2", i, c)
    await asyncio.sleep(i / 5)
    print("<U2", i, c)
    return("U2", i, c)    

results = batch_runner([user_func1, user_func2])
for result in results:
    print(result)

answered Feb 12, 2017 at 10:52

Udi

30.8k9 gold badges105 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Slakker Over a year ago

Thank you! Its a great feeling to see a working solution that is several lines shorter than my original question. :) One more question: If the user_funcs had a **kwargs as the 3rd positional argument, how to maintain context of that as well? (No manipulation necessary, but needs to passed in during batch_runner(...)).

Udi Over a year ago

func(*args, **kwargs) ?

Collectives™ on Stack Overflow

Python: Separating asyncio coroutine declaration from event looping/execution

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related