Run commands in parallel in bash script

Question

I have a script.sh file something like this:

python3 utility.py query-1.json 
python3 utility.py query-2.json
python3 utility.py query-3.json
.
.

I am currently using sh script.sh to run the script, this however runs each command in a sequential order. I am new to the concepts of parallelism, I would really appreciate any pointers to study more about it in this context, also is there any bash way of achieving multiprocessing or parallelism here ?

Put your jobs in the background with & (gnu.org/software/bash/manual/html_node/Job-Control.html). — Poshi
– Poshi, Commented Oct 10, 2022 at 13:40

ramsay · Accepted Answer · 2022-10-10 15:28:15Z

3

You could ask each statement to run in the background which could achieve what you want:

python3 utility.py query-1.json&
python3 utility.py query-2.json&
python3 utility.py query-3.json&
...
wait

The wait command makes the script run in the foreground until all background tasks have completed.

If you want more control, for example run one statement per CPU core, you could take a look at GNU parallel:

cat script.sh | parallel

answered Oct 10, 2022 at 15:28

ramsay

3,9553 gold badges17 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Run commands in parallel in bash script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related