Spawning multiple python scripts in bash

Question

I use the below script to spawn multiple python scripts in the server. The argument given to each of the script l is different and varies from 0 to 150.

for i in `seq 0 9`;
do 
    for j in `seq 0 14`;
    do 
        f=$((i*15))
        l=$((j+f))
        ./script --l ${l} &
    done
    i=0
    for job in `jobs -p`
    do
        pids[${i}]=${job}
        i=$((i+1))
    done
    for pid in ${pids[*]}; do
        wait ${pid}
    done 
done

The above syntax spawns 15 python scripts and waits for each of the child processes to finish before spawning the subsequent 15. However, the python script takes little time for some values of l and lots of time for other values of l. My problem is that, if for a given set of 15 l values, the script takes very little time for 14 of them and lots of time for one value of l, then the above script waits for the one script to finish before spawning the next 15, which results in many of the cores in the server remaining idle. I was wondering if there is a way to modify the above code to spawn a new child process every time one of them finishes (so that there are 15 child processes running at any given time).

Thanks

Have you considered using the multiprocessing library in python? docs.python.org/3/library/multiprocessing.html — iggy12345
– iggy12345, Commented Sep 18, 2019 at 4:44

Amadan · Accepted Answer · 2019-09-18 04:52:01Z

1

This should do the trick:

for i in $(seq 150)
do
  while [[ $(jobs | wc -l) -ge 15 ]]
  do
    wait -n $(jobs -p)
  done
  script --l "$i" &
done

The idea is that if there are 15 or more jobs, wait till at least one of them finishes till there's not more than 15 running; then spawn a next one. wait -n is useful as it will stop waiting when any of the mentioned processes exits.

edited Sep 18, 2019 at 4:52

answered Sep 18, 2019 at 4:46

Amadan

200k23 gold badges252 silver badges321 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spawning multiple python scripts in bash

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related