Bash (parallel wait execution) [duplicate]

Question

I've got a master script that reads 5 lines from a file and executes 5 distinct shell scripts at a time and waits for 5 to finish before starting the next batch:

cat file |
while read line1 && read line2  && read line3 && read line4 && read line5 
sh script1.sh &
sh script2.sh &
sh script3.sh &
sh script4.sh &
sh script5.sh &
wait

Is there a way to ensure that there are always 5 scripts running at a time even if scripts {1,3} complete. ie if 1-5 are running concurrently and 1,2 finish. Start 6,7 immediately.

...or the GNU extension xargs -P, if you prefer C to Perl. — Charles Duffy
– Charles Duffy, Commented Nov 29, 2017 at 1:56
line{1..5} are assigned as values for subsequent schema keys. ie schema1 = line1 and so on.... I then take these variables and build SQL output which i later execute for each distinct table. — manawa
– manawa, Commented Nov 29, 2017 at 15:42

John Moon · Accepted Answer · 2017-11-29 01:50:48Z

0

Interesting question! The best way I can think of to do it is to keep an array of the processes' PIDs and spool up a new one if there are fewer than five in the array. Something like this:

while read line1 && read line2  && read line3 && read line4 && read line5 ; do
    sh script1.sh &
    PIDS+=($!)
    sh script2.sh &
    PIDS+=($!)
    sh script3.sh &
    PIDS+=($!)
    sh script4.sh &
    PIDS+=($!)
    sh script5.sh &
    PIDS+=($!)

    for PID in "${PIDS[@]}"; do
        if ! kill -0 "$PID"; then # If the PID is not alive
            if [ ${#PIDS[@]} -lt 5 ]; then
                echo "Process $PID dead, starting new one"
                sh scriptN.sh &
                PIDS+=($!)
            fi
        fi
    done
done

It still needs some work, but hopefully that gives you the idea. GNU Parrallel wouldn't be a bad call either if that's available.

answered Nov 29, 2017 at 1:50

John Moon

9246 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Charles Duffy Over a year ago

boo, hiss re: all-caps variable names; see pubs.opengroup.org/onlinepubs/9699919799/basedefs/…, fourth paragraph, for relevant POSIX-specified conventions. (All-caps names are used for variables with meaning for the shell and OS; names with at least one lower-case character are reserved for application use and guaranteed not to conflict).

Charles Duffy Over a year ago

More on topic, consider wait -n (in bash 4.3 or later) to wait for exactly one process to exit without needing to iterate through the lot of 'em with kill -0 to figure out exactly which one it is.

Charles Duffy Over a year ago

(Also, this question is very much a FAQ -- we get a new instance of it once a month at least).

Charles Duffy Over a year ago

...personally, I'm pretty fond of Chepner's answer at stackoverflow.com/a/38775799/14122, using wait -n with prompt-style expansion to retrieve the number of jobs in-line.

John Moon Over a year ago

Thanks, Charles! I had thought all caps variables were conventional, but you've shown me the light. wait -n is also helpful here. Also, I didn't realize this was a FAQ - I've just recently started answering questions on here. Thanks for the feedback, cheers!

Collectives™ on Stack Overflow

Bash (parallel wait execution) [duplicate]

1 Answer 1

5 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Linked

Related