Python async await with timed loop

Question

I have a Python program that requests data from several API's then runs some machine learning over the data with intensive performance. I'm trying to find a way to keep the loop going, so it runs consistently every 5 mins, and the API gathering functions are run asynchronously, while the machine learning on that data is run once all the API calls have returned the data.

So, I have something which very basically looks like this:

    while(True):
      try:
        api_data_one = loadFromAPIOne()
        api_data_two = loadFromAPITwo()
    
        results = calculateResults(api_data_one, api_data_two)
        
      except Exception as e:
        print('Exception raised - Uable to complete this iteration', e)
    
      time.sleep(300)

Now because the machine learning takes about a minute or longer to complete, the iterations are taking longer than 5 mins to restart. And the API calls take some time on their own, so running them at the same time would be ideal.

Not having much experience with async and await I am struggling to find where to start. Could you please advise me on where to start?

Why do you want to do this with async/await in the first place? The concurrency of what you describe is at most 3, which is perfectly in the domain of what Threads can handle. Especially if your functions are blocking (as currently the API and calculate ones are), even an async event loop will run them in threads anyway. — MisterMiyagi
– MisterMiyagi, Commented Oct 11, 2020 at 11:00
I didn't even consider using Threads. They seem like a great option. Thanks will give it a try. — Jack -
– Jack -, Commented Oct 12, 2020 at 7:42

Gemini15 · Accepted Answer · 2020-10-11 14:28:26Z

To use async/await, for most languages, you have to define an asynchronous function (that you want to wait for) and then use await to emphasis the use of the newly created async function. When you define an async function in Python (using asyncio library), it is defined as a coroutine object which supports different operations from the asyncio library. I've not tested the following steps so I'm not really sure but you can start by,

defining loadFromAPIOne() and loadFromAPITwo as a async function, for ex :

async def loadFromAPIOne(): 
    # diverse operations like adding elements to an array 
    # or multiplying the array and so on

Same for loadFromAPITwo() and calculateResults()
Then you create another asynchronous function that will be run by asyncio, you can copy for example the following:

async def loopCalculus(): # eventually the parameters you need for the load function 
    while True: 
        try:
            api_data_one_task = asyncio.create_task(loadFromAPIOne())
            api_data_two_task = asyncio.create_task(loadFromAPITwo())
            
            api_data_one = await api_data_one_task
            api_data_two = await api_data_two_task

            results_task = asyncio.create_task(calculateResults(api_data_one, api_data_two))
            results = await results_task
            # and returning the results 
            return results
    
        except Exception as e:
            print('Exception raised - Uable to complete this iteration', e)


# and do not forget to run the async function by :
asyncio.run(loopCalculus())

# or easier creating an async main 
async def main():
    res = await loopCalculus()

You can see also the complete documentation :docs async await python Have a good day

Collectives™ on Stack Overflow

Python async await with timed loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related