1

I have a solver doing some calculations,one takes 5-6 hours. I start it with subprocess.Popen(). I have written a class starting one calculation and managing the solution process (class is operating as long as calculation is in progress). I have 40 cores for the whole problem. I decided not to calculate them in series assigning all 40 cores for each, but asynchronously using AsyncIO, 4 tasks at the time assigning 10 coreas for each. After anyone of these 4 tasks gets solved I want to start another one out of the whole number of tasks, so each moment of time I want 4 tasks being calculated untill all of them done. I have moddeled the situation by a simple code.

Here I have a coroutine doing some work which is aimed to model calculation process. I create event loop assigning only 2 tasks. When both are finished the code finishes. But I want it not to finish but create another task with random (for example) argumen and add it to task list. I am too noobie to understand how...

import asyncio


async def counter(n):
    print(f'counter with argument {n} has been launched')
    for i in range(n):
        for j in range(n):
            for k in range(n):
                pass
    await asyncio.sleep(0)
    print(f'counter with argument {n} has FINISHED')


my_loop = asyncio.get_event_loop()
tasks = [my_loop.create_task(counter(100)), my_loop.create_task(counter(500))]
wait_tasks = asyncio.wait(tasks)
my_loop.run_until_complete(wait_tasks)
my_loop.close()
6
  • Does this answer your question? how to add a coroutine to a running asyncio loop? Commented Nov 9, 2022 at 7:58
  • Is that multiprocessing? Commented Nov 9, 2022 at 7:59
  • asyncio doesn't distribute work among cores, it only does everything in a single thread ... you should be using multiprocessing or concurrent.futures.processpoolexecutor ... but your question seems to be about both asyncio tasks and multiprocessing ... you might want to make that more clear ... like add an explain what portion of the code is distributed to 4 tasks and which is distributed to 10 cores. Commented Nov 9, 2022 at 8:02
  • sorry, i didnt specified that. I assign numb of processes tp use for one task solving in bat file. When I Popen the bat file my OS launches calculation and assigns the specified proc numb in bat. So the code I am trying to write is just a kind of manager, managing one solution. But I want to create copies of this manager (operation in one core. like concurency) managing different tasks. Commented Nov 9, 2022 at 8:10
  • seems like you just want to run 4 tasks at the same time .... you can use the answer here to limit the number of tasks being executed, but as you are running an external bat file you should be using asyncio subprocess also see limit the number of tasks at the same time Commented Nov 9, 2022 at 8:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.