How can you write a while loop to run parallel processes on Unix ssh?

Question

I have a script file that contains different case IDs of a factorial Design of Experiment (DoE). I also have a script file run on MATLAB. I am attempting to run multiple cases of this DoE at once, 25 to be exact, and only up to 25 at once. The current input to the ssh server looks like the following (the batch file):

InputFile=inputfile.txt
count=0
while read -r case; do
echo $case
nohup matlab -nodisplay -nodesktop -r "try;runinput=$case;matlab_script;end;exit"
((++count % 25 == 0)) && wait
done <"$InputFile"

This however, only reads the first line of the inputfile.txt and runs that, and does not run the other 24 processes at the same time. I have tried getting rid of the && wait part but it still doesn't do what I want. How can I make it start the 24 other jobs in parallel?

Mark Setchell · Accepted Answer · 2021-02-22 09:11:10Z

1

And alternative to GNU Parallel, there's GNU xargs with its -P option:

xargs -P25 -d'\n' -I{} -t matlab -nodisplay -nodesktop -r "try;runinput={};matlab_script;end;exit" < inputfile.txt

In bash you could use wait -n to wait for a single child, something along:

count=0
while IFS= read -r case; do
    echo $case
    matlab -nodisplay -nodesktop -r "try;runinput=$case;matlab_script;end;exit" &
    if ((++count % 25 == 0)); then 
          if ! wait -n; then
               echo "ERROR - matlab failed!"
          fi
    fi
done <"$InputFile"
wait

edited Feb 22, 2021 at 9:11

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

answered Feb 22, 2021 at 9:05

KamilCuk

146k8 gold badges84 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

alexmesa Over a year ago

So how would my batch file look like? Do I run the first command or the second example you posted?

alexmesa Over a year ago

Another question, for both of these example, will the result when using top be 25 different jobs? Or is it treated as a single job?

KamilCuk Over a year ago

Do I run the first command or the second example you posted? One or the other. Sorry, but nothing will save you from understanding it. Read xargs documentation to understand it's options and learn basic bash scripting - research how to read from file line by line in bash, how to run processes in the background and how to wait for processes. will the result when using top be 25 different jobs? The processes are run in the background. There will be like 26 processes - 25 running running matlab and one running the shell or xargs. Research what is a process vs a thread.

KamilCuk Over a year ago

my batch file look like? The script is written in bash shell. A "batch" execution means an non-interactive execution - I have no idea if matlab reads input, the intention is to have non-interactive execution. Also .BAT windows cmd scripts are called "batch" scripts - I have no experience in windows CMD and this is not a .BAT script.

alexmesa Over a year ago

For the second script: if there are already 25 processes running from another similar script, does that mean that it won't run any processes? For some reason I had two of these command running at the same time, using different matlab scripts. one of them was already running, where the limit was 5 files, and ran another one, where the limit was 6. This resulted in only one process of the new batch command running at that instant, whilst the other 5 where still running from the old batch script.

|

Mark Setchell · Accepted Answer · 2021-02-22 09:01:20Z

0

I don't see where the ssh aspect of your code is, nor do I use Matlab, but if you want to keep 25 instances running at a time, I would suggest GNU Parallel, something like this:

parallel --dry-run -j25 -a inputfile.txt matlab -nodisplay -nodesktop -r "try;runinput={};matlab_script;end;exit"

answered Feb 22, 2021 at 9:01

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

4 Comments

alexmesa Over a year ago

I dont have access to the parallel command in my machine unfortunately.

Mark Setchell Over a year ago

It's a Perl script that anyone can install in 10 seconds... oletange.blogspot.com/2013/04/why-not-install-gnu-parallel.html

alexmesa Over a year ago

I am accessing a remote server that has multiple cores available. I do not own this server, therefore I do not have the rights to install anything.

Mark Setchell Over a year ago

AFAIK, you don't need to install GNU Parallel on your remote, multi-core host in order to run jobs on it from your local machine using locally-installed GNU Parallel. I believe the only requirement is that Perl exists on the remote machine. Maybe @OleTange will confirm this? Or you could try it quite easily.

Collectives™ on Stack Overflow

How can you write a while loop to run parallel processes on Unix ssh?

2 Answers 2

10 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related