2

I am using fastapi with beanie as ODM and asyncio, 2 functions taking a lot of time so I decided to run these function in parallel, I tried thread but it's giving a function running in a different event loop error. I tried multiple solutions but they are not working. Some solutions suggested that threading should not be used with asyncio.

asyncio.to_thread(add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id))
asyncio.to_thread(make_safe_dict_for_mongo_insertion_for_nmap(body.ip,inserted_id))

and other solutions are

asyncio.gather(
    add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id),
    make_safe_dict_for_mongo_insertion_for_nmap(body.ip, inserted_id)
)

and

asyncio.run(add_attack_shodan_results_to_scans_collection(cve_list_with_scan_id))
asyncio.run(make_safe_dict_for_mongo_insertion_for_nmap(body.ip,inserted_id))

and

thrd1 = threading.Thread(target=make_thread_for_cve_calculation, args=(cve_list_with_scan_id))
thrd1.start()
thrd = threading.Thread(target=make_thread_for_nmap, args=(body.ip, inserted_id))
thrd.start()

and threading functions are

def make_thread_for_nmap(ip,scan_id):
    logger.info(f'thread id:{threading.current_thread().ident}')
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(make_safe_dict_for_mongo_insertion_for_nmap(ip,scan_id))
    loop.close()

def make_thread_for_cve_calculation(*calculate_cve):
    logger.info(f'thread id:{threading.current_thread().ident}')
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    loop.run_until_complete(add_attack_shodan_results_to_scans_collection(calculate_cve))
    loop.close()

What is the solution to run these function parallel or background? I also have checked the documentation but did not find any solution. Is there any way to run these function parallel without blocking other requests and what is the solution for "different event loop" error?

12
  • Why do you think you need to combine threads and async? Commented Jul 30, 2023 at 19:31
  • What are your functions doing ? Are they doing IO operations, or calculation ? It is very complicated to help you without knowing the functions, Commented Jul 30, 2023 at 19:33
  • 2
    If you want to run them in threads, they likely shouldn't be async at all. Commented Jul 31, 2023 at 8:25
  • 1
    If you want two CPU-intensive calculations running in parallel, you don't want asyncio or threads, you want to use multiprocessing. (C)Python still has a so called Global Interpreter Lock, which means that "threading" is not about parallel processing, it can only do parallel waiting (for I/O, and that's why it's not really beneficial to mix it with asyncio). Commented Jul 31, 2023 at 9:18
  • 1
    You might find this answer and this answer helpful Commented Aug 1, 2023 at 5:05

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.