asyncio - functions run sequentially instead of concurrently [duplicate]

Question

I have a function that sends 2 different requests. I need to call this function with different parameters 20 times.

I would like to run the functions concurrently (different arguments) to spare some time between request and response.

This is a very simplified function:

async def get_data(url):
    return requests.get(url)

And this is how I call it:

loop = asyncio.get_event_loop()
tasks = [asyncio.ensure_future(get_data(url)) for url in websites.split('\n')]
group = asyncio.gather(*tasks)
results = loop.run_until_complete(group)
print(results)
loop.close()

The problem is that it runs sequentially instead of concurrently.

It's obvious that I'm missing something. Do you know what to do?

Your code is going to run sequentially because requests isn't asyncio-aware. You'll want to use a library like aiohttp so that additional requests can be made while you're awaiting others. — dirn
– dirn, Commented Oct 9, 2020 at 19:03

GProst · Accepted Answer · 2020-10-09 18:17:34Z

2

Don't wrap coroutine in asyncio.create_task, use * to unpack coroutins when passing to asyncio.gather and call loop.run_until_complete in the end

loop = asyncio.get_event_loop()
tasks = [get_data(url) for url in websites.split('\n')]
group = asyncio.gather(*tasks)
results = loop.run_until_complete(group)
print(results)
loop.close()

Plus, it won't be concurrent because requests isn't asynchronous, it's blocking the thread, you need to use alternative async HTTP clients like aiohttp.

edited Oct 9, 2020 at 18:17

answered Oct 9, 2020 at 18:04

GProst

10.3k3 gold badges28 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Milano Over a year ago

Thanks, this works but the problem is that it runs sequentially instead of concurrently.

GProst Over a year ago

What makes you think so?

Milano Over a year ago

I've added print(url) inside the function and it prints one after another with 1 - 3 second pauses.

GProst Over a year ago

Updated with a note

Collectives™ on Stack Overflow

asyncio - functions run sequentially instead of concurrently [duplicate]

1 Answer 1

4 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Linked

Related