Asynchronous Background task in python with asyncio library

Question

I'm looking to run a heavy computing task in the background without blocking IO. Problem here is my main function does not depend on the heavy task and needs to return value before/while executing the heavy computing task. As an example:

def main(args):
   transformed_data_list:List[Dict] = translate_request_to_object(args)
   status = insert_data_into_db(transformed_data:Dict)
   if(status)
      # running background task
      asyncio.run(process_background_task(transformed_data_list))

   # Here, I want to return a success response as soon as data inserted into db
   return "data insert into db"

   async process_background_task(transformed_data_list:List[Dict]):
      for data in transformed_data_list:List:
         asyncio.create_task(heavy_computation_task(data))

But above code not returning response until process_background_task completed.

stuporuser · Accepted Answer · 2024-07-08 01:09:55Z

6

How to run task in background

asyncio.run is the blocking function that starts event loop. If you want to start process_background_task in background you need to use asyncio.create_task and make main asynchronous. Then run asyncio.run(main(...)).

async def main(args):
    transformed_data_list:List[Dict] = translate_request_to_object(args)
    status = insert_data_into_db(transformed_data:Dict)
    if status:
        # running background task
        asyncio.create_task(process_background_task(transformed_data_list))

    # Here, I want to return a success response as soon as data inserted into db
    return "data insert into db"


async def process_background_task(transformed_data_list:List[Dict]):
    for data in transformed_data_list:List:
        asyncio.create_task(heavy_computation_task(data))

# Start event loop, execute task and wait until task finish.
asyncio.run(main(...))

How to run heavy tasks in event loop

But in the case heavy_computation_task will block event loop you need to use ProcessPoolExecutor with loop.run_in_executor.

Here is the example from documentation:

import asyncio
import concurrent.futures

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    # Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())

In your case:

async def main(args):
    transformed_data_list:List[Dict] = translate_request_to_object(args)
    status = insert_data_into_db(transformed_data:Dict)
    if status:
        # running background task
        asyncio.create_task(process_background_task(transformed_data_list))

    # Here, I want to return a success response as soon as data inserted into db
    return "data insert into db"


async def process_background_task(transformed_data_list:List[Dict]):
    loop = asyncio.get_running_loop()

    for data in transformed_data_list:List:
        with concurrent.futures.ProcessPoolExecutor() as pool:
            await loop.run_in_executor(pool, heavy_computation_task, data)

# Start event loop, execute task and wait until task finish.
asyncio.run(main(...))

edited Jul 8, 2024 at 1:09

stuporuser

32 bronze badges

answered Jun 9, 2022 at 17:32

Oleg Utkin

1811 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

MathProblem Over a year ago

Could you please why you suggest ProcessPoolExecutor instead of ThreadPoolExecutor ? And from where I call asyncio.run(main(...)) will again block that parent function, right?

Oleg Utkin Over a year ago

ThreadPoolExecutor can run simultaneously only one task, so it only suits IO-bound operations, but not CPU-bound operations. ProcessPoolExecutor can run many tasks simultaneously.

Oleg Utkin Over a year ago

asyncio.run(main(...)) will block the function where it runs.

AXO May 10 at 13:43

As noted in the docs, you should probably save a reference to the result of create_task function, to avoid task disappearance mid-execution due to garbage collection.

Collectives™ on Stack Overflow

Asynchronous Background task in python with asyncio library

1 Answer 1

How to run task in background

How to run heavy tasks in event loop

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

How to run task in background

How to run heavy tasks in event loop

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related