Bash: Output columns from array consisting of two columns

Question

Problem

I am writing a bash script and I have an array, where each value consists of two columns. It looks like this:

for i in "${res[@]}"; do
  echo "$i"
done

#Stream1
0 a1
1 b1
2 c1
4 d1
6 e1
#Stream2
0 a2
1 b2
3 c2
4 d2
9 f2
...

I would like to combine the output from this array into a larger table, and multiplex the indices. Furthermore, I would like to format the top row by inserting comment #Sec. I would like the result to be something like this:

#Sec  Stream1 Stream2
0     a1      a2
1     b1      b2
2     c1
3             c2
4     d1      d2
6     e1
9             f2

The insertion of #Sec and removal of the # behind the Streamkeyword is not necessary but desired if not too difficult.

Tried Solutions

I have tried piping to column and awk, but have not been able to produce the desired results.

EDIT

resis an array in a bash script. It is quite large, so I will only provide a short selection. Running echo "$( typeset -p res)"produces following output:

declare -a res='([1]="#Stream1
0 3072
1 6144
2 5120
3 1024
5 6144
..." [2]="#Stream2
0 3072
1 5120
2 4096
3 3072
53 3072
55 1024
57 2048")'

As for the 'result', my initial intention was to assign the resulting table to a variable and use it in another awk script to calculate the moving averages for specified indices, and plot the results. This will be done for ~20 different files. However I am open to other solutions.

The number of streams may vary from 10 to 50. Each stream having from 100 to 300 rows.

what do you mean by 'result'? are you just looking to print to stdout? or are you looking to populate a new bash array where each entry in the array is one of your lines of output? also, could you update the question with the output from typeset -p res (this will allow us to verify the structure of the array as well as easily recreate the array in our environments) — markp-fuso
– markp-fuso, Commented Sep 28, 2021 at 15:18
how many streams can there be? while your examples show just 2x streams, the last line of your 'input' is ... which seems to indicate (to me) that there may be more than 2x streams — markp-fuso
– markp-fuso, Commented Sep 28, 2021 at 16:38
Good question! Realizing, formatting will be off I will edit my post based on your questions. — nela
– nela, Commented Sep 28, 2021 at 16:38
anubhava's answer appears to work for a variable number/size of streams so at a minimum you could pipe your for/echo loop into his awk solution; generally speaking ... awk excels at parsing (multiple) input files and generating pretty much any output you can imagine ... so I'm thinking it may be possible to have just one (awk) script replace the 'several scripts' you mention ... — markp-fuso
– markp-fuso, Commented Sep 28, 2021 at 17:26
I've recently started using awk rather than python scripts for parsing, and I realize it is very powerful. Looking forward to learning more about it! — nela
– nela, Commented Sep 28, 2021 at 18:09

anubhava · Accepted Answer · 2021-09-28 15:14:47Z

2

You may use this awk solution:

cat tabulate.awk

NF == 1 {
   h = h OFS substr($1, 2)
   ++numSec
   next
}
{
   keys[$1]
   map[$1,numSec] = $2
}
END {
   print h
   for (k in keys) {
      printf "%s", k
      for (i=1; i<=numSec; ++i)
         printf "\t%s", map[k,i]
      print ""
   }
}

Then use it as:

awk -v OFS='\t' -v h='#Sec' -f tabulate.awk file

#Sec  Stream1  Stream2
0     a1       a2
1     b1       b2
2     c1
3              c2
4     d1       d2
6     e1
9              f2

edited Sep 28, 2021 at 15:14

answered Sep 28, 2021 at 15:07

anubhava

790k67 gold badges603 silver badges671 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

nela Over a year ago

Thanks for the clever solution! It works like a charm! I have another question, however. Where would I need to apply the change in order to substitute empty field values with zeros?

anubhava Over a year ago

Use

awk -v OFS='\t' -v h='#Sec' 'NF==1{h = h OFS substr($1,2); ++numSec; next} {keys[$1]; map[$1,numSec] = $2} END {print h; for (k in keys) {printf "%s", k; for (i=1; i<=numSec; ++i) printf "\t%s", ((k,i) in map ? map[k,i] : "0"); print ""}}' file

nela Over a year ago

Man, thats a pretty elegant way to do it. Thanks for the help

Collectives™ on Stack Overflow

Bash: Output columns from array consisting of two columns

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related