Python async request call inside loop

Question

I'm trying to develop a Python API with FastAPI, and I have to download a series of images from several URLs, in a foor loop.

The problem is that I have to retrieve the images, and I can't do it as it is an async operation inside a normal for loop.

Here is my code:

@app.post("/v1/img_color")
async def img_color(request: Request):
    body = await request.json()
    images = []
    
    for img in body['images']:
        img_id = img['id']
        img_url = img['url']
        r = requests.get(img_url)
        content = await r.content
        dw_image = Image.open(BytesIO(content))
        images.append(dw_image)
    

    return images

But it gives me the following error:

TypeError: object bytes can't be used in 'await' expression

How can I solve this? I've searched about this issue, and found some solutions regarding asyncio but I can't manage to make them work.

Update:

Following the suggestion of deleting the await code from await r.content, I've reached to another error, which says the following:

TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have dict attribute')

I don't quite understand this, as I can retrieve perfectly the url from the JSON body of the original POST request...

Traceback

Traceback (most recent call last):
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 388, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\applications.py", line 190, in __call__
    await super().__call__(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\middleware\errors.py", line 181, in __call__
    raise exc from None
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\middleware\errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\exceptions.py", line 82, in __call__
    raise exc from None
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\starlette\routing.py", line 41, in app
    response = await func(request)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\routing.py", line 196, in app
    response_data = await serialize_response(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\routing.py", line 124, in serialize_response

    return jsonable_encoder(response_content)
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 102, in jsonable_encoder
    jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 140, in jsonable_encoder
    return jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 88, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 88, in jsonable_encoder
    encoded_value = jsonable_encoder(
  File "d:\users\kendo\documents\git\empathy\kala_api\env\lib\site-packages\fastapi\encoders.py", line 139, in jsonable_encoder
    raise ValueError(errors)
ValueError: [TypeError('cannot convert dictionary update sequence element #0 to a sequence'), TypeError('vars() argument must have __dict__ attribute')]

requests.get() is synchronous. There is nothing to await. When the call is done, the content bytes are completely available. Remove the await. — Tomalak
– Tomalak, Commented Dec 5, 2020 at 23:08
Does this answer your question? How could I use requests in asyncio? — Tomalak
– Tomalak, Commented Dec 5, 2020 at 23:08
@Tomalak I've deleted the await code, but now it gives me a new error (see the update), any idea? Thanks in advance! — Pablo Cañal Suárez
– Pablo Cañal Suárez, Commented Dec 6, 2020 at 12:14
Since I have no idea on what line that error occurs, I can't help you. — Tomalak
– Tomalak, Commented Dec 6, 2020 at 12:18
I'm guessing: You do return images, which is a list of Image objects. I imagine your framework is configured to try to convert this to JSON, and it seems to try to turn your [Image1, Image2] into an object {"key1": "value1", "key2": "value2"}. That can't work because there aren't any keys in there. You can try to return enumerate(images) to see if the error goes away. You'll need to figure out what return output you want from your endpoint, and what you need to return from your handler function to get it. — Tomalak
– Tomalak, Commented Dec 6, 2020 at 12:41

Tomalak · Accepted Answer · 2020-12-06 13:11:19Z

1

requests.get() is synchronous. There is nothing to await. When the call is done, the content bytes are completely available. Remove the await.

for img in body['images']:
    r = requests.get(img['url'])
    dw_image = Image.open(BytesIO(r.content))
    images.append( (img['id'], dw_image) )

If you're feeling adventurous (or want to learn about dict comprehensions), you can replace your entire loop with one line

@app.post("/v1/img_color")
async def img_color(request: Request):
    body = await request.json()
    return {img['id']: Image.open(BytesIO(requests.get(img['url']).content)) for img in body['images']}

edited Dec 6, 2020 at 13:11

answered Dec 6, 2020 at 12:47

Tomalak

339k68 gold badges547 silver badges635 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python async request call inside loop

Update:

Traceback

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Update:

Traceback

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related