Multiprocessing in Shell Script (BASH)

Question

I've written a script and What I'm wanting to do is get it to run as fast as possible by using sub-processing or background-processing or what ever I need to use in order to speed up the computations.

So I was looking at using the & symbol in my script to simply speed up processing wherever I could, but when I put it in here, I don't get the results echo'ed to me.

This is the function that I have written and am trying to improve apon.

InTri(){
    while read line
    do
        V1X=$(echo "$line" | awk '{print $1}') &
        V1Z=$(echo "$line" | awk '{print $2}') &
        V2X=$(echo "$line" | awk '{print $3}') &
        V2Z=$(echo "$line" | awk '{print $4}') &
        V3X=$(echo "$line" | awk '{print $5}') &
        V3Z=$(echo "$line" | awk '{print $6}') &
        run=$(echo "$line" | awk '{print $7}') &
        wait
        echo "$V1X $V1Z $V2X $V2Z $V3X $V3Z $run"
    done < <(mysql -u root -ppassword LightCycle -N -e "SELECT V1X, V1Z, V2X, V2Z, V3X, V3Z, ID FROM Temp3 WHERE (V1X <= $x OR V2X <= $x OR V3X <= $x) AND (V1X >= $x OR V2X >= $x OR V3X >= $x) AND (V1Z <= $z OR V2Z <= $z OR V3Z <= $z) AND (V1Z >= $z OR V2Z >= $z OR V3Z >= $z);")
}

I've read the "man bash" (or what of it I could understand) but I don't understand why this is not working.

Help! :)

Thanks to anyone who can, your input is much appreciated.

What do you want to DO with these variables once you've collected them? — ghoti
– ghoti, Commented Nov 21, 2012 at 23:08

TolMera · Accepted Answer · 2012-11-20 11:29:27Z

2

You might think that creating more processes will make it faster, but it might actually make it slower because of the overhead of creating new processes and all that piping.

Why not simply do the following?

mysql -u root -ppassword LightCycle -N -e "SELECT V1X, V1Z, V2X, V2Z, V3X, V3Z, ID FROM Temp3 WHERE (V1X <= $x OR V2X <= $x OR V3X <= $x) AND (V1X >= $x OR V2X >= $x OR V3X >= $x) AND (V1Z <= $z OR V2Z <= $z OR V3Z <= $z) AND (V1Z >= $z OR V2Z >= $z OR V3Z >= $z);" | awk '{print $1" "$2" "$3" "$4" "$5" "$6" "$7}'

Or if you want to read them into variables:

while read V1X V1Z V2X V2Z V3X V3Z run
do
  echo "$V1X $V1Z $V2X $V2Z $V3X $V3Z $run"
done < <(mysql -u root -ppassword LightCycle -N -e "SELECT V1X, V1Z, V2X, V2Z, V3X, V3Z, ID FROM Temp3 WHERE (V1X <= $x OR V2X <= $x OR V3X <= $x) AND (V1X >= $x OR V2X >= $x OR V3X >= $x) AND (V1Z <= $z OR V2Z <= $z OR V3Z <= $z) AND (V1Z >= $z OR V2Z >= $z OR V3Z >= $z);")

edited Nov 20, 2012 at 11:29

TolMera

41610 silver badges25 bronze badges

answered Nov 20, 2012 at 10:52

dogbane

276k77 gold badges407 silver badges415 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

TolMera Over a year ago

Thanks your input sped this up by "real 0m0.793" :) And I learned something :) great

Michael Krelin - hacker · Accepted Answer · 2012-11-20 10:27:15Z

2

Whatever happens in & line happens in a child process, that can't modify parent's environment. You'll probably need to open pipes and read from them in parent process. For tasks with comparable costs of execution with your example it isn't worth the effort.

In this example you can just do

cat <(echo "$line" | awk '{print $1}') <(echo "$line" | awk '{print $1}') ...

Instead of this whole loop body.

answered Nov 20, 2012 at 10:27

Michael Krelin - hacker

144k25 gold badges200 silver badges177 bronze badges

3 Comments

TolMera Over a year ago

Ok, but using a pipe each process waits for its predecessor to finish before it starts. I'm processing 26000 triangles for a total of 480billion points, any small speed increase would be a large speed increase over the number of computations that this is running. And what you see here, is only a fraction of the code thats actually running, it was just the best sample for me to use for this question.

Michael Krelin - hacker Over a year ago

Whether anyone waits for anyone depends on how you do it. For instance, I see no reason for waiting in the code above. One can test it, of course. As for your particular task, I'm not sure bash is the right tool.

Michael Krelin - hacker Over a year ago

@TolMera, try this to test: cat <(echo a >&2; sleep 2 ; echo b >&2) <(echo c >&2 ; sleep 3 ; echo d >&2)

Guru · Accepted Answer · 2012-11-20 10:33:30Z

2

If you feel your program is getting slower due to those 7 awk commands, you can do something like this without any awk:

$ line="1 2 3 4 5 6 7"
$ a=($line)
$ echo ${a[0]}
1
$ echo ${a[1]}
2
$ echo ${a[@]}
1 2 3 4 5 6 7

where "a" is an array which contains all the elements of the line variable in it.

answered Nov 20, 2012 at 10:33

Guru

17.1k2 gold badges37 silver badges47 bronze badges

Comments

TolMera · Accepted Answer · 2012-11-20 11:15:02Z

In order to get the script above working as it should, I just needed to remove the & before the "wait"

InTri(){
    while read line
    do
        V1X=$(echo "$line" | awk '{print $1}') &
        V1Z=$(echo "$line" | awk '{print $2}') &
        V2X=$(echo "$line" | awk '{print $3}') &
        V2Z=$(echo "$line" | awk '{print $4}') &
        V3X=$(echo "$line" | awk '{print $5}') &
        V3Z=$(echo "$line" | awk '{print $6}') &
        run=$(echo "$line" | awk '{print $7}') 
        wait
        echo "$V1X $V1Z $V2X $V2Z $V3X $V3Z $run"
    done < <(mysql -u root -ppassword LightCycle -N -e "SELECT V1X, V1Z, V2X, V2Z, V3X, V3Z, ID FROM Temp3 WHERE (V1X <= $x OR V2X <= $x OR V3X <= $x) AND (V1X >= $x OR V2X >= $x OR V3X >= $x) AND (V1Z <= $z OR V2Z <= $z OR V3Z <= $z) AND (V1Z >= $z OR V2Z >= $z OR V3Z >= $z);")
}

Collectives™ on Stack Overflow

Multiprocessing in Shell Script (BASH)

4 Answers 4

1 Comment

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related