1

On an individual node I have been able to run the precompiled function update_OSS_internal_compiler in parallel (16 cores) with different inputs specified by $FileCounter. However, I would like to extend this parallel processing to beyond one node to multiple nodes, but I'm not sure how to approach it.

#!/bin/bash

. /u/local/Modules/default/init/modules.sh
module load matlab
export MCR_CACHE_ROOT=$TMPDIR

Macro_Iter=10
ApertNum=121

FullPath=$(pwd)
TempFileFolder=$FullPath/TempFiles

for MacroLoop in $(seq 1 1 $Macro_Iter); do
# WANT TO SSH INTO DIFFERENT NODES AND RUN SAME PROCESS WITH DIFFERENT INPUTS WHILE UPDATING FILECOUNTER AFTER EACH NODE, OR DO SOMETHING SIMILAR
   for FileCounter in $(seq 1 1 $ApertNum); do echo $FileCounter; done | xargs -I{} --max-procs 16 bash -c '
        {
        echo "doing aperture {}"
        ./update_OSS_internal_compiler {}
        } '
      done
   done

   echo "$FullPath/TempFiles/ApertFiles"
   ./update_OSS_global_compiler

Any help is appreciated.

1
  • One idea would be to use the hostnames hostnames=cat $PE_HOSTFILE|awk '{print $1}' to ssh into individual nodes, and run processes in background, then ssh into another node to do the same, etc. Commented Jan 29, 2016 at 1:26

1 Answer 1

1

Check if you are reinventing GNU Parallel:

parallel -S worker1 -S worker2 ./update_OSS_internal_compiler ::: arg1 arg2 arg3

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling

Installation

If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.