Using a loop variable in a Bash script to pass different command-line arguments

Question

I have a C++ program to which I pass two doubles as inputs from the command line using

int main(int argc, char *argv[]){
    double a,b;
    a = atof(argv[1]);
    b = atof(argv[2]);
    further code.....

I run the code on a cluster using the qsub utility and I have a Bash script named 'jobsub.sh` to submit the jobs which looks like this:

#!/bin/csh -f
hostname
cd /home/roy/codes/3D             # Change directory first -- replace Mysubdir
set startdir = `pwd`               # Remember the directory we're in
if( ! -d /scratch/$USER ) then
    mkdir /scratch/$USER       # Create scratch directory
endif                              # If it does not exist
#cp infile12 /scratch/$USER     # Copy input file to scratch directory
cd /scratch/$USER                  # Change to scratch directory
#rm *.*
$HOME/codes/3D/autoA100.out 2.1 2.2          # Run a program
cp * $startdir         # Copy outputfiles back to where we started

At the terminal I do qsub jobsub.sh.

However, I want to run the same executable for different values of a and b in parallel on different cores. Is it possible to write a for loop in the Bash script so that I can do something like,

for i=1;i<=10;i++ {
   $HOME/codes/3D/autoA100.out 2+i*0.1 2+i*0.2
}

Your "bash" script resembles bash, but contains many syntax errors. — jordanm
– jordanm, Commented Jul 6, 2012 at 1:21
@jordanm: In some ways it looks like csh (set for example). — Dennis Williamson
– Dennis Williamson, Commented Jul 6, 2012 at 2:05
@DennisWilliamson I edited the script in the post, it actually is a csh — lovespeed
– lovespeed, Commented Jul 6, 2012 at 12:13
Your question is tagged bash, you refer to "bash" in the text and your script is named jobsub.sh (but it's actually csh). Do you want answers in Bash or Csh? By the way, Csh Programming Considered Harmful. — Dennis Williamson
– Dennis Williamson, Commented Jul 6, 2012 at 12:29

chrisaycock · Accepted Answer · 2012-07-06 00:26:40Z

3

If you are submitting the execution script to a batch loader, then there is no way to have a simple loop do the execution like you want because the entire script is run on each node. However, most batch loaders provide environment variables to the user.

PBS, for example, has $PBS_ARRAYID, which specifies the unique ID for the job as it is running. So instead of using a loop, your script can have:

a=$(echo "2+($PBS_ARRAYID+1)*0.1" | bc -l)
b=$(echo "2+($PBS_ARRAYID+1)*0.2" | bc -l)
$HOME/codes/3D/autoA100.out $a $b

Notice I've added 1 to $PBS_ARRAYID above because the ID begins from 0 while your loop begins from 1. It's also worth mentioning that while bash can do some arithmetic natively, it cannot handle real numbers; that's why I had to invoke bc.

answered Jul 6, 2012 at 0:26

chrisaycock

38.1k15 gold badges94 silver badges128 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

jordanm Over a year ago

It can handle some real numbers, integers. It can not do floating point math.

forest.peterson Over a year ago

@chrisaycock sorry to revive this old post but there has never been an accepted solution; you said " there is no way to have a simple loop do the execution like you want because the entire script is run on each node." How is this solved using two scripts, one for the loop and one for qsub?

Jo So · Accepted Answer · 2012-07-06 01:39:49Z

0

You can always run jobs in the background (in which case the shell will continue with the next instruction)

MY_PIDS=
for i in 1 2 3 4 5 6 7 8 9 10
do
    yourjob "$(a i)" "$(b i)" &   # Note the ampersand (background)
    MY_PIDS="$MY_PIDS "$!         # Do not write the $! inside the quotes, bash will
                                  # break (better, don't use bash but plain sh)
done

# Wait wor each job to complete and cleanup
for p in $MY_PIDS
do
    wait "$p"
    echo "Return code of job $p is $?"
    do_cleanup_for "$p"
done

But certainly you need to make sure that the jobs running in parallel are not stepping on each other's feet (like writing to the same file).

edited Jul 6, 2012 at 1:39

answered Jul 6, 2012 at 0:26

Jo So

26.9k6 gold badges45 silver badges60 bronze badges

5 Comments

jordanm Over a year ago

Quoting $! is only a problem in interactive shells.

Jo So Over a year ago

Enough of a problem to not quote it above, I'd say.

jordanm Over a year ago

Is there a reason you used a string to store pids instead of an array?

Dennis Williamson Over a year ago

Bro, the OP tagged the question bash.

Jo So Over a year ago

What's wrong? Works in bash, too. Easy to understand too, though probably not the answer to OP's question ;-)

Collectives™ on Stack Overflow

Using a loop variable in a Bash script to pass different command-line arguments

2 Answers 2

2 Comments

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related