Process multiple element using asyncio instead of threads

Question

I have a program that uses python threads like this: starting 5 threads and start processing a list of elements.

def process_element(element):
   print(element)

jobs = Queue()
    
def do_stuff(q):
    while not q.empty():
        value = q.get()
        process_element(element=value)
        q.task_done()
 
for i in line: # my list of element
    jobs.put(i)
    
for i in range(5):
    worker = threading.Thread(target=do_stuff, args=(jobs))
    worker.start()

jobs.join()

How can I use asyncio to do the same work.

when i try to migrate to asyncio i don't know how to translate my code to it — salah mazroai
– salah mazroai, Commented Dec 28, 2020 at 11:49

user4815162342 · Accepted Answer · 2020-12-28 11:42:56Z

1

A literal translation of your code into asyncio would look like this:

import asyncio, random

async def process_element(element):
    print('starting', element)
    await asyncio.sleep(random.random())  # simulate IO-bound processing
    print('done', element)

async def do_stuff(q):
    while not q.empty():
        value = await q.get()
        await process_element(element=value)
        q.task_done()

async def main():
    jobs = asyncio.Queue()

    for i in range(20):
        await jobs.put(i)

    for i in range(5):
        asyncio.create_task(do_stuff(jobs))

    await jobs.join()

asyncio.run(main())

Note, however, that:

Asyncio handles IO-bound tasks such as communicating with HTTP servers or remote databases, chat servers, etc. It doesn't handle CPU-bound tasks, where a long-running operation will block the whole event loop. (In such cases multithreading or multiprocessing are more appropriate.) For this reason "converting" multithreaded code to asyncio often fails.
while not q.empty(): ... process ... is an anti-pattern in both threading and asyncio because it doesn't allow waiting for something to arrive in the queue. If you know all your items in advance, you don't need a queue in the first place, you could use ordinary lists.

answered Dec 28, 2020 at 11:42

user4815162342

159k22 gold badges350 silver badges418 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

salah mazroai Over a year ago

yes I know all my element before starting the process

Collectives™ on Stack Overflow

Process multiple element using asyncio instead of threads

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related