running each element in array in parallel in bash script

A lot of lenghty and theoretical answers here, I'll try to keep it simple - what about using | (pipe) to connect the commands as usual ?;) (And GNU parallel, which excels for these type of tasks).

seq 6 | parallel -j4 "command --arg1 {} | command2 > results/{}"

The -j4 will limit number of threads (jobs) as requested. You DON'T want to write to a single file from multiple jobs, output one file per job and join them after the parallel processing is finished.

Using GNU Parallel it looks like this:

array=( 1 2 3 4 5 6 )
parallel -0 --bar --tagstring '{= $_=localtime(time)."\t".$_; =}' \
  command --arg1 {} ::: "${array[@]}" > output

GNU Parallel makes sure output from different jobs is not mixed.

If you prefer the output from jobs mixed:

parallel -0 --bar --line-buffer --tagstring '{= $_=localtime(time)."\t".$_; =}' \
  command --arg1 {} ::: "${array[@]}" > output-linebuffer

Again GNU Parallel makes sure to only mix with full lines: You will not see half a line from one job and half a line from another job.

It also works if the array is a bit more nasty:

array=( "new
line" 'quotes"  '"'" 'echo `do not execute me`')

Or if the command prints long lines half-lines:

command() {
  echo Input: "$@"
  echo '"  '"'"
  sleep 1
  echo -n 'Half a line '
  sleep 1
  echo other half

  superlong_a=$(perl -e 'print "a"x1000000')
  superlong_b=$(perl -e 'print "b"x1000000')
  echo -n $superlong_a
  sleep 1
  echo $superlong_b
}
export -f command

GNU Parallel strives to be a general solution. This is because I have designed GNU Parallel to care about correctness and try vehemently to deal correctly with corner cases, too, while staying reasonably fast.

GNU Parallel guards against race conditions and does not split words in the output on each their line.

array=( $(seq 30) )
max_proc_count=30

command() {
  # If 'a', 'b' and 'c' mix: Very bad                                                         
  perl -e 'print "a"x3000_000," "'
  perl -e 'print "b"x3000_000," "'
  perl -e 'print "c"x3000_000," "'
  echo
}
export -f command

parallel -0 --bar --tagstring '{= $_=localtime(time)."\t".$_; =}' \
  command --arg1 {} ::: "${array[@]}" > parallel.out

# 'abc' should always stay together
# and there should only be a single line per job
cat parallel.out | tr -s abc

GNU Parallel works fine if the output has a lot of words:

array=(1)
command() {
  yes "`seq 1000`" | head -c 10M
}
export -f command

parallel -0 --bar --tagstring '{= $_=localtime(time)."\t".$_; =}' \
  command --arg1 {} ::: "${array[@]}" > parallel.out

GNU Parallel does not eat all your memory - even if the output is bigger than your RAM:

array=(1)
outputsize=1000M
export outputsize
command() {
    yes "`perl -e 'print \"c\"x30_000'`" | head -c $outputsize
}
export -f command

parallel -0 --bar --tagstring '{= $_=localtime(time)."\t".$_; =}' \
  command --arg1 {} ::: "${array[@]}" > parallel.out

You know how to execute commands in separate processes. The missing part is how to allow those processes to communicate, as separate processes cannot share variables.

Basically, you must chose whether to communicate using regular files, or inter-process communication/FIFOs (which still boils down to using files).

The general approach :

• Decide how you want to present tasks to be executed. You could have them as separate files on the filesystem, as a FIFO special file that can be read from, etc. This could be a simple as writing to a separate file each command to be executed, or writing each command to a FIFO (one command per line).

• In the main process, prepare the files describing tasks to perform or launch a separate process in the background that will feed the FIFO.

• Then, still in the main process, launch worker processes in the background (with &), as many of them as you want parallel tasks being executed (not one per task to perform). Once they have been launched, use wait to, well, wait until all processes are finished. Separate processes cannot share variables, you will have to write any output that needs to be used later to separate files, or a FIFO, etc. If using a FIFO, remember more than one process can write to a FIFO at the same time, so use some kind of mutex mechanism (I suggest looking into the use of mkdir/rmdir for that purpose).

• Each worker process must fetch the next task (from a file/FIFO), execute it, generate the output (to a file/FIFO), loop until there are no new tasks, then exit. If using files, you will need to use a mutex to "reserve" a file, read it, and then delete it to mark it as taken care of. This would not be needed for a FIFO.

• Depending on the case, your main process may have to wait until all tasks are finished before handling the output, or in some cases may launch a worker process that will detect and handle output as it appears. This worker process would have to either be stopped by the main process once all tasks have been executed, or figure out for itself when all tasks have been executed and exit (while being waited on by the main process).

This is not detailed code, but I hope it gives you an idea of how to approach problems like this.

(Community Wiki answer with the OP's proposed self-answer from the question -- now edited out):

So here is one way I can think of doing this, not sure if this is the most efficient way and also, I can't control the amount of threads (I think, or processes?) this would use:

array=( 1 2 3 4 5 6 )

lag_func () {
  echo "$1"

  lags=($(command --arg1 $1)

  lngth_lags=${#lags[*]}

  for (( i=1; i<=$(( $lngth_lags -1 )); i++))
  do
    result=${lags[$i]}
    echo -e "$timestamp\t$result" >> $log_file
    echo "result piped"
  done
}

for each in "${array[@]}"
do
  lag_func $each &
done