How to use asyncio with django rest api framework just the way jquery promises work so that responses don't need to wait

Question

I am on python 3.7 with django 2.2.3 running. I want a solution with asyncio so that the api can just call the async function & return the response without waiting just the way we do things with the jquery promises. the definition my_coro is just for example. I will be running moviepy functions that usually require 40-50 seconds to complete. I don't want the api to wait that long for sending out the response. I am also confused on how to handle the thread pool. how to use thread pool here? because I intend to make the moviepy iterations fast as well. So how to create a pool for handling my_coro calls?

async def my_coro(n):
    print(f"The answer is {n}.")

async def main(): 
    await asyncio.gather(my_coro(1),my_coro(2),my_coro(3),my_coro(4))

class SaveSnaps(APIView):
    def post(self, request, format = None):
        if request.user.is_anonymous:
            return Response({"response":"FORBIDDEN"}, status = 403)
        else:
            try:
                asyncio.run(main())
                return Response({"response": "success"}, status = 200)
            except Exception as e:
                return Response({'response':str(e)}, status = 400)

Update : I tried using celery. but as I won't be using periodic tasks & actually the method I need to make asynchronous receives a blob array as parameter. celery's task.delay is giving me an error because tasks expect serializable params. So I am back to square one on this. I am not sure whether I should stick to the threading solution or something else.

Update : I forgot to share what I did at last. I shifted to celery. but since celery's task.delay expect serialized params, I moved the blob saving part to a synchronous method which after completion, hands over the moviepy tasks to the celery run task.

Can you elaborate more on the data you are passing? What do you mean with Blob Array? — Radwan Abu-Odeh
– Radwan Abu-Odeh, Commented Jan 19, 2020 at 7:19
I meant a FormData object containing 1000-1200 png blob objects — Ashitaka
– Ashitaka, Commented Apr 28, 2020 at 4:18

Radwan Abu-Odeh · Accepted Answer · 2020-01-15 10:12:41Z

6

As far as I know, asyncio is going to process your code concurrently, but your application is still going to wait for the execution of asyncio to be done, then it continues execution inside your app.

If you want your app to run the code in the background and continue execution, similar to Promises in JS, you need to consider using Job Scheduling using Celery for example, or something similar.

This is a simple Guide for using Django with Celery.

answered Jan 15, 2020 at 10:12

Radwan Abu-Odeh

2,06511 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ashitaka Over a year ago

Ok I am going for celery after all. thanks for the tip

ramsey0 · Accepted Answer · 2021-03-01 14:55:26Z

I've been struggling with this issue, and this is what I've found.

The compatibility issues between DRF and asyncio are mostly at the Router layer of DRF.

If you override the as_view in a ViewSet you will run into errors where the Router is expecting either a callable or a list of callables.

This means there are 2 categories of solutions:

Skip / Customize DRF routers.
Wrap the routers URLPattern.

This is what I've had the most success with so far, perhaps others can provide improvements on it.

router = OptionalSlashRouter()

router.register( ... )


def _async_wrapper(func):
    async def wrapped(*args, **kwargs):
        loop = asyncio.get_event_loop()
        kwargs['loop'] = loop
        return await sync_to_async(func)(*args, **kwargs)
    return wrapped


urlpatterns = [
    url(urlpattern.pattern,
        _async_wrapper(urlpattern.callback),
        name=urlpattern.name)
    for urlpattern in router.urls
]

I previously tried just wrapping the callables in the sync_to_async function, but this didn't seem to allow the views themselves to get the event loop.

However, with this pattern, I was able to call loop.create_task(some_coroutine) from inside a drf viewset.

There are a lot of downsides to this pattern, of course. For one thing, you have to pass the event loop around if you want to use it.

I may end up just removing the drf routers entirely. But this is obviously a drastic step in most projects as there are often a lot of urls managed by DRF routers.

so even if I use asyncio.run(async_method) within my view it will block the asgi event loop, right?

Collectives™ on Stack Overflow

How to use asyncio with django rest api framework just the way jquery promises work so that responses don't need to wait

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related