python for each run async function without await and parallel

Question

I have 10 links in my CSV which I'm trying to run all at the same time in a loop from getTasks function. However, the way it's working now, it send a request to link 1, waits for it to complete, then link 2, etc, etc. I want the 10 links that I have to run all whenever startTask is called, leading to 10 requests a second.

Anyone know how to code that using the code below? Thanks in advance.


import requests
from bs4 import BeautifulSoup
import asyncio

def getTasks(tasks):
    for task in tasks:
      asyncio.run(startTask(task))


async def startTask(task):
    
    success = await getProduct(task)
    if success is None:
    return startTask(task)

    success = await addToCart(task)
    if success is None:
    return startTask(task)

    ...
    ...
    ...

getTasks(tasks)

Artyom Vancyan · Accepted Answer · 2022-12-05 12:38:15Z

1

+50

First of all, to make your requests sent concurrently, you should use the aiohttp instead of the requests package that blocks I/O. And use the asyncio's semaphore to limit the count of concurrent processes at the same time.

import asyncio
import aiohttp

# read links from CSV
links = [
    ...
]

semaphore = asyncio.BoundedSemaphore(10) 
# 10 is the max count of concurrent tasks
# that can be processed at the same time.
# In this case, tasks are requests.

async def async_request(url):
    async with aiohttp.ClientSession() as session:
        async with semaphore, session.get(url) as response:
            return await response.text()


async def main():
    result = await asyncio.gather(*[
        async_request(link) for link in links
    ])
    print(result)  # [response1, response2, ...]


if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())
    loop.close()

answered Dec 5, 2022 at 12:38

Artyom Vancyan

5,4023 gold badges17 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sean Berrie Over a year ago

I need main to just call the function startTask(task) and that to be it. Then startTask(task) can recall itself the whole time as stated in the code I posted. I've been playing around with the code you posted but I'm pretty sure it doens't allow me to do that.

Artyom Vancyan Over a year ago

You just described a recursion, but I would not use recursion there as it is going to be run 24/7 and it will require huge resources. just use while True and wrap in try to make it work after errors.

Sean Berrie Over a year ago

Decides to just do it like this and "return" if success is None. Works fine now even though I'm pretty sure its not the best way to do it but its fine for my use case. ``` while True: await asyncio.gather(*[ startTask(task) for task in tasks ]) time.sleep(5) ```

Collectives™ on Stack Overflow

python for each run async function without await and parallel

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related