How to read table-like file into multiple arrays (bash)

Question

If I have a file example1.txt containing multiple strings

str1
str2
str3
...

I can read them into a bash array by using

mapfile -t mystrings < example1.txt.

Now say my file example2.txt is formatted as a table

str11 str12 str13
str21 str22 str23
str31 str32 str33
...   ...   ...

and I want to read each column into a different array. I know I can use other tools such as awk to separate each line into fields. Is there some way to combine this functionality with mapfile? I'm looking for something like

mapfile -t firstcol < $(cat example2.txt | awk '//{printf $1"\n"}')
mapfile -t secondcol < $(cat example2.txt | awk '//{printf $2"\n"}')

(which doesn't work).

Any other suggestion on how to handle a table in bash is also welcome.

chepner · Accepted Answer · 2015-08-28 20:47:07Z

6

Reading each row is simple, so let's build off that. I'll assume you have a proper matrix (i.e., each row has the same number of columns. This will be much easier since you are using bash 4.3.

while read -a row; do
    c=0
    for value in "${row[@]}"; do
        declare -n column=column_$(( c++ ))
        column+=( "$value" )
    done
done < table.txt

There! Now, did it work?

$ echo "${column_0[@]}"
str11 str21 str31 
$ echo "${column_1[@]}"
str12 str22 str32

I think so!

declare -n makes a nameref to an array (implicitly declared by the += on the next line) using a counter that increments as we iterate over each row. Then we simply append the current column value to the array behind the current nameref.

edited Aug 28, 2015 at 20:47

answered Aug 28, 2015 at 20:43

chepner

538k77 gold badges594 silver badges746 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Charles Duffy Over a year ago

This is almost the approach I went with myself (didn't yet have the answer on which version to target when I chose something different).

anubhava · Accepted Answer · 2015-08-28 20:40:15Z

2

You should be using process substitution like this:

mapfile -t firstcol < <(awk '{print $1}' example2.txt)

mapfile -t secondcol < <(awk '{print $2}' example2.txt)

mapfile -t thirdcol < <(awk '{print $3}' example2.txt)

answered Aug 28, 2015 at 20:40

anubhava

790k67 gold badges603 silver badges671 bronze badges

6 Comments

Charles Duffy Over a year ago

Seems quite inefficient to be reading the file once per column rather than doing a single pass.

anubhava Over a year ago

@mbrandalero: process substitution acts like a file and avoids creating temporary files. @Charles: Yes, if there are large number of columns then yes it will involve running awk that many times. My answer was focussing to fix the problem in OP's attempt of mapfile with awk.

Charles Duffy Over a year ago

@mbrandalero, if you did a pipe from awk to mapfile, the data would be lost when the subshell running mapfile on the right-hand side of the pipeline exited. Thus, using this idiom is necessary.

chepner Over a year ago

Unless the file is huge, this is probably a good tradeoff for versions prior to 4.3. Getting my approach to work without namerefs is a bit tedious.

mbrandalero Over a year ago

Thanks for the useful explanation!

|

Charles Duffy · Accepted Answer · 2015-08-28 20:52:24Z

1

Hmm. Something like this, perhaps?

readarrays() {
  declare -a values
  declare idx line=0
  while read -a values; do
    for idx in "${!values[@]}"; do
      [[ ${@:idx+1:1} ]] || break
      declare -g "${@:idx+1:1}[$line]=${values[@]:idx:1}"
    done
    (( ++line ))
  done
}

Tested as:

bash4-4.3$ (readarrays one two three <<<$'a b c\nd e f'; declare -p one two three)
declare -a one='([0]="a" [1]="d")'
declare -a two='([0]="b" [1]="e")'
declare -a three='([0]="c" [1]="f")'

edited Aug 28, 2015 at 20:52

answered Aug 28, 2015 at 20:43

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Collectives™ on Stack Overflow

How to read table-like file into multiple arrays (bash)

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related