1

I have a for loop in which a function task is called. Each call to the function returns a string that is appended to an array. I would like to parallelise this for loop. I tried using & but it does not seem to work.

Here is the code not parallelised.

task (){ sleep 1;echo "hello $1"; }
arr=()

for i in {1..3}; do
    arr+=("$(task $i)")
done

for i in "${arr[@]}"; do
    echo "$i x";
done

The output is:

hello 1 x
hello 2 x
hello 3 x

Great! But now, when I try to parallelise it with

[...]
for i in {1..3}; do
    arr+=("$(task $i)")&
done
wait
[...]

the output is empty.

Improve this question
0

4 Answers 4

Reset to default
4

GNU Parallel is good at doing stuff in parallel :-)

task (){ sleep 1;echo "hello $1"; }

# Make "task" known to sub shells
export -f task

# Do tasks in parallel
parallel -k task ::: {1..3}

Sample Output

hello 1
hello 2
hello 3

I am suggesting you do - but Charles kindly points out that this is a known bash pitfall:

array=( $(parallel -k task ::: {1..3}) )

Charles' suggested solution is:

IFS=$'\n' read -r -d '' -a array < <(parallel -k task ::: 1 2 3 && printf '\0')
Improve this answer
Sign up to request clarification or add additional context in comments.

14 Comments

Executing you code yields zsh:1: command not found: task. Is it a zsh-specific issue?
@BiBi, you tagged this with bash, not zsh. Please use correct tags for your setup.
Just added the zsh tag. I'm also interested in bash, I'm going to try your solution with bash.
export -f is a bash-ism that makes functions visible to sub-shells
IFS=$'\n' read -r -d '' -a array < <(parallel -k task ::: 1 2 3 && printf '\0') -- note that I stopped using {1..3} because of some bugs in how old versions of bash handle it. values=( {1..3} ) and then parallel -k task ::: "${values[@]}" sidestep those issues.
|
0

Try encapsulating your adding step into a function or a temporary script, but send the items to a file instead. I think you will need some command that I can't remember on top of my head for dealing with file lock. See if you need to export things too. Then you source the file at the end. Something like:

echo 'arr=(' > temp
add() { echo item >> temp; }
...
export -f add
...
add &
...
echo ')' > temp
source temp
rm temp
Improve this answer

2 Comments

You're going to need to fill in the ...s more to have a correct solution -- at minimum, for example, you'll need to wait for all the subprocesses to finish. And you'll need to be more careful to not accidentally truncate the file. (Though one thing you don't need to worry about here is file locking, so long as each thread ends in a single write of less-than-a-page length with the O_APPEND flag set).
I would strongly suggest having successfully tested an answer at least once (copied-and-pasted, not manually entered with human-length delays between commands) before calling it complete.
0

You are looking for parset (part of GNU Parallel since 20170422) or env_parset (available since 20171222):

# If you have not run:
#    env_parallel --install
# and logged in again, then you can instead run this to activate (env_)parset:
. `which env_parallel.bash`

task (){
  echo "hello $1"
  sleep 1.$1
  perl -e 'print "binary\001\002\n"'
  sleep 1.$1
  echo output of parallel jobs do not mix
}
env_parset arr task ::: {1..3}
env_parset a,b,c task ::: {1..3}

echo "${arr[1]}" | xxd
echo "$b" | xxd

parset is supported in Bash/Ksh/Zsh (including arrays), ash/dash (without arrays).

Improve this answer

Comments

-2

Try encapsulating your adding step into a function or a temporary script. See if you need to export things too. Something like:

add() { arr+=(value); }
...
export -f add
...
add &
Improve this answer

1 Comment

This doesn't work, because background tasks are run in a subprocess; they can't impact the variables that are active in the parent-process shell.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.