I have a solver doing some calculations,one takes 5-6 hours. I start it with subprocess.Popen(). I have written a class starting one calculation and managing the solution process (class is operating as long as calculation is in progress). I have 40 cores for the whole problem. I decided not to calculate them in series assigning all 40 cores for each, but asynchronously using AsyncIO, 4 tasks at the time assigning 10 coreas for each. After anyone of these 4 tasks gets solved I want to start another one out of the whole number of tasks, so each moment of time I want 4 tasks being calculated untill all of them done. I have moddeled the situation by a simple code.
Here I have a coroutine doing some work which is aimed to model calculation process. I create event loop assigning only 2 tasks. When both are finished the code finishes. But I want it not to finish but create another task with random (for example) argumen and add it to task list. I am too noobie to understand how...
import asyncio
async def counter(n):
print(f'counter with argument {n} has been launched')
for i in range(n):
for j in range(n):
for k in range(n):
pass
await asyncio.sleep(0)
print(f'counter with argument {n} has FINISHED')
my_loop = asyncio.get_event_loop()
tasks = [my_loop.create_task(counter(100)), my_loop.create_task(counter(500))]
wait_tasks = asyncio.wait(tasks)
my_loop.run_until_complete(wait_tasks)
my_loop.close()
multiprocessingorconcurrent.futures.processpoolexecutor... but your question seems to be about both asyncio tasks and multiprocessing ... you might want to make that more clear ... like add an explain what portion of the code is distributed to 4 tasks and which is distributed to 10 cores.