Being more explicit with processing in parallel with python in parallel

Question

I am attempting to create a python script that can drive multiple MPI simulations (F90 executables, though it doesn't matter). Each of these MPI simulations use 2 processors. Lets say I want to have three of these MPI simulations running simultaneously. If I run these 3 simulation from the command line in 3 separate terminals, without python, they each get their own 2 processors, and run as though they are the only things that exist in the world.

My current implementation does not appear to be doing this. It is clear from tracking the MPI simulations that there is competition amongst the MPI simulations. Here is my current procedure

import subprocess
import multiprocessing as mp

def execute(inputs, output):
    do_stuff_with_inputs()
    subprocess.call('mpiexec -np 2 my_executable.x', shell=True)
    results = post_process_stuff()
    output.put(results)


output = mp.Queue()
processes = []
for i in xrange(3):
    process.append(mp.Process(target=execute, args=args)))

for p in process:
    p.start()

for p in process:
    p.join()

results = [output.get() for p in process]

What I would like to do is be more explicit with the procedure, somehow 'creating' processor space in python so that the executable call has its own dedicated number of processors.

Also, the posted code doesn't define args and seems to just run 2 execute subprocesses. Also, what is your evidence that "It is clear ... that there is competition amongst the MPI simulations"? — Tom Dalton
– Tom Dalton, Commented Mar 15, 2018 at 16:47
Using mpi4py, you can spawn processes using sub_comm = MPI.COMM_SELF.Spawn('slavef90', args=[], maxprocs=1). See my answer stackoverflow.com/questions/41699585/… You can even communicate to and from the spawned processes, but that requires altering the fortran code accordingly. — francis
– francis, Commented Mar 15, 2018 at 18:07
@TomDalton I have detailed outputs from my_executable.x that determine how efficiently the simulation ran, these outputs show that using the method above is twice as slow as running 2 simulations at the same time without python in separate terminals from the command line. And yes, I did not include args, the code snippet was not intended to be able to produce anything, since one would need 'my_executable.x' as well. Also the subprocess are the whole point of the procedure, since I need the subprocesses to complete in order get the results list at the end and continue to do more stuff — Kschau
– Kschau, Commented Mar 15, 2018 at 18:12
How does my_executable.x process input and produce output? What do do_stuff_with_inputs and post_process_stuff do? Regarding the subprocesses, I was confused because you said "Lets say I want to have three of these MPI simulations running simultaneously." but your given code only seems to run 2. — Tom Dalton
– Tom Dalton, Commented Mar 15, 2018 at 21:30

Kschau · Accepted Answer · 2018-03-16 17:00:45Z

0

For my purposes, the suggestion from @GillesGouaillarde was sufficient, removing CPU binding from my subprocess call

subprocess.call('mpiexec -bind-to none -np 2 my_executable.x', shell=True)

sped up performance to an acceptable level.

answered Mar 16, 2018 at 17:00

Kschau

1561 silver badge14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Being more explicit with processing in parallel with python in parallel

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related