Bash Filling Array in Parallel

Question

Trying to stop using files output to store data from a parallel for loop. I want to run the output of array through paralleled commands using input from the array and store the results when its completed in the array in the correct order.

Logic-ish code:

    for i in `do some stuff`
    do

    bunch of stuff based on variables in the loop run in parallel but output saved in the right order

    done

Broken Code:

    #!/usr/bin/bash
    IFS=$'\n'
    for i in $(seq 1 1 10)
    do

    export MYBROKENARRAY+=(`echo POTATO;wait`) &  

    done

    sleep 1
    echo MYBROKENARRAY IS "${MYBROKENARRAY[@]}"

if I keep the & I get my parallel but the output is not stored in the array as expected

expedited output is:

MYBROKENARRAY IS POTATO POTATO POTATO POTATO POTATO POTATO POTATO POTATO POTATO POTATO

This is how I to it today with ugly file output ... :(

    #!/usr/bin/bash
    IFS=$'\n'
    for i in $(seq 1 1 10)
    do

    echo POTATO > out_${i}.txt &

    done
    cat out_*.txt

Create an array of N elements and give each child process a number X between 0 and N-1. Each process will fill the array on entry X. — Bayou
– Bayou, Commented Sep 10, 2019 at 13:15
You have a concurrency problem that is dealt with by database engines, using transactions and atomic operations. It is possible to emulate this by having a separate process that receive entries and is in charge of adding them to the array. It will prevent concurrent destructive accesses to the array by two different processes. — Léa Gris
– Léa Gris, Commented Sep 10, 2019 at 13:22
for i in `do some stuff` is almost certainly wrong; see Bash FAQ 001. — chepner
– chepner, Commented Sep 10, 2019 at 13:49

Mark Setchell · Accepted Answer · 2019-09-10 13:24:37Z

1

You can use GNU Parallel like this:

MY_SHOUTY_WORKING_ARRAY=( $(parallel -k echo POTATO ::: {1..10} ))

Output

echo "${MY_SHOUTY_WORKING_ARRAY[*]}"
POTATO 1
POTATO 2
POTATO 3
POTATO 4
POTATO 5
POTATO 6
POTATO 7
POTATO 8
POTATO 9
POTATO 10

answered Sep 10, 2019 at 13:24

Mark Setchell

210k32 gold badges309 silver badges503 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ole Tange · Accepted Answer · 2019-09-11 14:33:56Z

0

GNU Parallel has parset which is made for this:

parset my_array -j10 'sleep 10; seq {}' ::: {1..10}
echo "${my_array[3]}"

This also deals nicely with output containing special characters such as newlines.

You activate parset with:

env_parallel --install
<<start a new shell>>

edited Sep 11, 2019 at 14:33

answered Sep 10, 2019 at 16:24

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Collectives™ on Stack Overflow

Bash Filling Array in Parallel

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related