parse .csv data into matrix or 2 dimensional array bash/shell awk

Question

I have a coma delimited csv file named 'itrs.csv' which I want to parse into a matrix or 2d array using a script bash or shell

Loads\PostDate,schedule,seta,eeta,2019-11-05,2019-11-06,2019-11-07,2019-11-08
BANAMEX,7,1:18:10,1:23:45,G,G,C,C
EMEA,5,0:21:00,1:01:00,G,G,G,C

I have tried the following:

declare -A matrix
eval matrix=($(awk -f, itrs.csv))
for ((i=0;i<=2;i++))
do
    for ((j=0;j<=$6;j++))
    do
    echo ${matrix[$i,$j]}" " 
    done
    echo
done

but above code is throwing errors. I also would like to know how to check the number of columns and rows while parsing data because csv file size may change.

but above code is throwing errors And what are those errors? — Shawn
– Shawn, Commented Nov 12, 2019 at 0:29
Take a look at the awk man page and then think about what -f, means. Also see why-is-using-a-shell-loop-to-process-text-considered-bad-practice for some of the problems with your approach. — Ed Morton
– Ed Morton, Commented Nov 12, 2019 at 16:50

KamilCuk · Accepted Answer · 2019-11-12 00:48:42Z

2

Well, you can do this: Create an associative array, iterate over lines and keep the count of the current line, then iterate over the fields and create an associative array with indexes as requested.

i=0
declare -A matrix
while IFS=, read -r -a line; do
   for ((j = 0; j < ${#line[@]}; ++j)); do
        matrix[$i,$j]=${line[$j]}
    done
    ((i++))
done < itrs.csv

After it declare -p matrix would output:

declare -A matrix=([1,5]="G" [1,4]="G" [1,7]="C" [1,6]="C" [1,1]="7" [1,0]="BANAMEX" [1,3]="1:23:45" [1,2]="1:18:10" [0,4]="2019-11-05" [0,5]="2019-11-06" [0,6]="2019-11-07"[0,7]="2019-11-08" [0,0]="Loads\\PostDate" [0,1]="schedule" [0,2]="seta" [0,3]="eeta" [2,6]="G" [2,7]="C" [2,4]="G"[2,5]="G" [2,2]="0:21:00" [2,3]="1:01:00" [2,0]="EMEA" [2,1]="5" )

See bashfaq How can I read a file (data stream, variable) line-by-line (and/or field-by-field)?
Don't use eval. eval is evil. Don't eval arr=($(..)) unless you know what you are doing. In your case, using eval looks like it has little to zero sense.
The error comes from awk. awk works like awk [options] script [file], you could awk -F, '{print $0}' itrs.csv, but it would make no sense. The itrs.csv is parsed by awk as being the script - as it makes no sense as a awk script, the tool throws an error.
To read for example the first line only separated by comma into an array in bash you can IFS=, line=($(head -n1 itrs.csv)). The -F, affects how awk parses the file, not how bash creates array - for that use IFS.

edited Nov 12, 2019 at 0:48

answered Nov 12, 2019 at 0:40

KamilCuk

146k8 gold badges84 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Elcappoo Over a year ago

Thank you. I appreciate your time. So in here, I understand how IFS is working but since I have access to each field within the csv, will I be able to change some fields ${line[$j]} directly in the csv after doing some validations? Or is it just read/printing access?

KamilCuk Over a year ago

I don't understand the question. The < itrs.csv opens an input stream, it doesn't output anything, it can't change the file contents. Editing such file would be easy in awk, without any associative array in bash. I don't understand if you want to edit the file (which is a different job then reading the file into an associative array...) or you want to edit the associative array like matrix[2,3]="new_value".

Elcappoo Over a year ago

I apologize for not being clear. what I want to do is loop through the fields in the csv file, do some validations and fire sql querries to get status then update csv field. I thought I will be able to achieve that by parsing data from csv to a matrix.

Ed Morton Over a year ago

@Elcappoo what you want to do is fine but you'd never use a shell for that, you'd just do it in awk or some other text processing tool. See why-is-using-a-shell-loop-to-process-text-considered-bad-practice

Ed Morton Over a year ago

Ask a question if you'd like help with that and I'm sure someone will be able to help you. Make sure to provide a minimal reproducible example in your question which includes concise, testable sample in[put and expected output.

|

Collectives™ on Stack Overflow

parse .csv data into matrix or 2 dimensional array bash/shell awk

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related