Parallel nested for loop in bash

Question

I am trying to run a c executable through bash. The executable will take a different argument in each iteration, and I want to do it in parallel since I have 12 cores available.

I tried

w=1;
for i in {1..100}
do
l=$(($i-1));
for j in {12*l..12*i}
do
./run $w/100 > "$w"_out &
done
expr=$w % 12;
if ["$expr" -eq "0"]
then wait;
fi;
done

run is the c executable. I want to run it with increasing argument w in each step, and I want to wait until all processes are done if 12 of the cores are in use. SO basically, I will run 12 executables at the same time, then wait until they are completed, and then move to the next 12.

Hope I made my point clear.

Cheers.

Is that {12*l..12*i} supposed to do arithmetic? A multiplication? You are not using that j in the loop. Is that intended? Is the $W % 12 supposed to perform math as well? Are you interested in error handling of all the different calls as well? — Alfe
– Alfe, Commented Jul 22, 2014 at 13:04
Are you aware that your variable w is never changed? It is just set to 1 in the beginning, and that's it. — Alfe
– Alfe, Commented Jul 22, 2014 at 13:09
Using expr as a variable name is likely to cause grief at some point... Some code indenting/formatting might make things a bit easier to understand. Add a set -x before your loop(s) to see exactly what is being run... — twalberg
– twalberg, Commented Jul 22, 2014 at 14:02

Ole Tange · Accepted Answer · 2014-07-22 23:14:24Z

3

Use gnu parallel instead:

parallel ./myscript {1} ::: {1..100}

You can specify the number of parallel processes with the -P option, but it defaults to the number of cores in the system.

You can also specify -k to keep the output order and redirect the file.

To redirect the output to individual files, you can specify the output redirection, but you have to quote it, so that it is not parsed by the shell. For example:

parallel ./run {1} '>' {1}_out ::: {1..10}

is equivalent to running ./run 1 > 1_out to ./run 10 > 10_out

edited Jul 22, 2014 at 23:14

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

answered Jul 22, 2014 at 13:34

user000001

33.7k14 gold badges86 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user178882 Over a year ago

Thank you. But, could you please be more specific? I am having quite hard times to learn parallel. From shell, I call ./run 1 > 1_out and I will increase 1's to some number. My outputs will be like 1_out, 2_out, 3_out (for 1,2,3)

user000001 Over a year ago

You can do that by quoting the redirection character. See edit for an example.

Ole Tange Over a year ago

@user178882 see if man parallel_tutorial helps you learn GNU Parallel.

Pranav Joglekar Over a year ago

Verify if your parallel version is correct. On Ubuntu 20, I already had a parallel executable installed, but it wasn't supporting the syntax above. Had to install the required executable seperately.

Collectives™ on Stack Overflow

Parallel nested for loop in bash

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related