I want to use a shell scrip in a for-loop that run 100 files in parallel.
Currently, I have a shell script of the following format:
#!/bin/bash
NUM=10
python a1.py $((NUM + 0)) &
python a2.py $((NUM + 2)) &
python a3.py $((NUM + 4)) &
python a4.py $((NUM + 6)) &
python a5.py $((NUM + 8)) &
Now, if I have a1.py, a2.py, a3.py .... a100.py, and I want to run them in parallel, how do I do it in for-loop?
If you have bash version 4 and run this:
echo {10..208..2}
You will get this:
10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60
62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 92 94 96 98 100 102 104 106 108
110 112 114 116 118 120 122 124 126 128 130 132 134 136 138 140 142 144 146
148 150 152 154 156 158 160 162 164 166 168 170 172 174 176 178 180 182 184
186 188 190 192 194 196 198 200 202 204 206 208
which looks like your series. Then, if you want to run lots of jobs in parallel, I would use GNU Parallel. That offers you {#} as a placeholder for the job number. So, if you run this:
parallel -k echo {#} {} ::: {10..208..2}
You will get this:
1 10
2 12
3 14
4 16
5 18
So, to run your actual scripts, something like:
parallel -k --dry-run 'python a{#}.py {}' ::: {10..208..2}
Sample Output
python a1.py 10
python a2.py 12
python a3.py 14
python a4.py 16
...
...
If that looks good, run again without the --dry-run and without the -k which keeps the output in order to make it easier to debug.
TLDR;
My most concise answer, with GNU Parallel is:
parallel python a{#}.py {} ::: {10..208..2}
Or if you don't have bash version 4:
parallel python a{#}.py {} ::: $(seq 10 2 208)
parallel to show the intermediate output of the python-script (I wrote some print statement and it is not showing up on the terminal).
parallel --line-buffer ... or look through the manual pages with man parallelNUM=10
for ((i=0; i<100; i++)); do echo python a$(($i+1)).py $(($NUM+$i*2)); done
Output:
python a1.py 10 python a2.py 12 python a3.py 14 . . . python a98.py 204 python a99.py 206 python a100.py 208
If this looks okay, use:
NUM=10
for ((i=0; i<100; i++)); do python a$(($i+1)).py $(($NUM+$i*2)) & done
