While read loop in parallel

Question

I have the following while loop in a bash script however I would like to run these in parallel (an failing) can anyone point me in to the right direction please?

Thanks!

while read LINE; do
    RAYID=$(echo "$LINE" | jq -r .rayId)
    LINE="$(echo $LINE | sed 's/\([[:digit:]]\{13\}\)[[:digit:]]\{6\}/\1/g')"
    args=( -XPUT "localhost:9200/els/logs/$RAYID?pipeline=geoip-els" -H "Content-Type: application/json" -d "$LINE" )
    curl "${args[@]}" > /dev/null 2>&1
done <<< "$ELS_LOGS"

Just add a & if you want to run each curl command in the background — Tom Fenech
– Tom Fenech, Commented Oct 3, 2017 at 12:56
Kudos for using jq and an array to store the arguments. I might suggest trying to move more of the processing into jq as well, so that you have something like echo "$ELS_LOGS" | jq '...' | while read -r rayid line; do args=(...); curl ...; done. — chepner
– chepner, Commented Oct 3, 2017 at 13:28

Matias Barrios · Accepted Answer · 2017-10-03 16:11:16Z

1

** EDITED

Additionally to what @TomFenech stated which is correct, I want to add that it would be also nice if you add wait after done, so the script won't finish its execution, until all tasks are completed.

function doSomething(){

        RAYID=$(echo "$1" | jq -r .rayId  ) 
        LINE="$(echo $1 | sed 's/\([[:digit:]]\{13\}\)[[:digit:]]\{6\}/\1/g'  )" 
        args=( -XPUT "localhost:9200/els/logs/$RAYID?pipeline=geoip-els" -H "Content-Type: application/json" -d "$1"  ) 
        curl "${args[@]}" > /dev/null 2>&1 
}

while read LINE; do
   doSomething $LINE &
done <<< "$ELS_LOGS"
wait

Regards!

edited Oct 3, 2017 at 16:11

answered Oct 3, 2017 at 13:18

Matias Barrios

5,0864 gold badges31 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Matthew Bullock Over a year ago

When I run the above I think it just tries to run all the lines at any time rather than one after the other as it gives me curl errors. :/

Matias Barrios Over a year ago

I have edited my response in order to fix you sync problem. Hope it helps you. I basically wrapped your four lines in a function but I did not test it. So let me know. @MatthewBullock

Matthew Bullock Over a year ago

This is why I love to hate Bash seems likes the function now ignores line '1' & '2' (JQ rayid and the SED) of the function

Matthew Bullock Over a year ago

Got It! had to replace $1 with $line. Thanks for the help @matias Barrios

Matias Barrios Over a year ago

@MatthewBullock No problem buddy.

Collectives™ on Stack Overflow

While read loop in parallel

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related