Extracting data from a column in a CSV file with headers and input into an array in Bash

Question

I am trying to write a simple bash script that can extract data from one column in csv file and store it as an array. My question is very similar to that of a previous post, but I am trouble getting the proposed solution from that post to work (possibly because my CSV file has headers).

Specifically, I have a CSV file, weights.csv, with two columns:

w_neg,w_pos
1.000,1.000
0.523,1.477
0.210,1.790
1.420,0.580

and I would like to create an array variable, w_pos, that will contain the entire second column of weights.csv.

w_pos=(1.000 1.477 1.790 0.580)

Based on the answer from this previous post, I tried to do this using the following line of code:

w_pos=( $(cut -d ',' -f2 weights.csv ) )

Unfortunately, it seems as if this only stores the first row of w_pos. As

echo ${w_pos[0]} 
1.000

but

echo ${w_pos[1]}

yields nothing.

I would appreciate any insight into what the problem might be. Ideally, I would like a solution that does not use packages other than what would be bundled with a barebones Unix installation (the script has to run on a cluster that doesn't have simple tools like "bc" :-/)

This works for me. Also, with your example file ${w_pos[0]} should be w_pos while ${w_pos[1]} should be 1.000 instead. What do you see for ${w_pos[*]}? — user2303197
– user2303197, Commented May 26, 2014 at 19:20

jaypal singh · Accepted Answer · 2014-05-26 19:35:52Z

2

Here is a way using bash:

while IFS=, read -r col1 col2; do
    [[ $col2 =~ ^[0-9] ]] && w_pos+=( $col2 )
done < weights.csv

declare -p w_pos

Output:

declare -a w_pos='([0]="1.000" [1]="1.477" [2]="1.790" [3]="0.580")'

We set the delimiter to , by modifying the IFS.
We then read the two columns in two variables
Append to w_pos array variable the second column, if the variable starts with a number. [[ $col2 =~ ^[0-9] ]] does that test for us.
declare -p will give you the structure of our array.

edited May 26, 2014 at 19:35

answered May 26, 2014 at 19:20

jaypal singh

77.6k24 gold badges108 silver badges147 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Vytenis Bivainis · Accepted Answer · 2014-05-26 19:13:01Z

1

Here's the solution:

w_neg=($(tail -n +2 weights.csv | cut -d, -f1))
w_pos=($(tail -n +2 weights.csv | cut -d, -f2))

answered May 26, 2014 at 19:13

Vytenis Bivainis

2,38622 silver badges29 bronze badges

2 Comments

Berk U. Over a year ago

This does not work unfortunately. Running echo ${w_neg[@]}$ returns a blank line. Could it be an issue with the csv file?!

Berk U. Over a year ago

Thanks! It turns out that original code is correct - but somehow does not work for CSV files that were created in MS Excel.

Md Shihab Uddin · Accepted Answer · 2020-05-07 08:33:31Z

0

i like to use awk. Here is an way you can try:

w_neg=($(tail -n +2 weights.csv | awk -F ',' '{print $1;}'))
w_pos=($(tail -n +2 weights.csv | awk -F ',' '{print $2;}'))
echo ${w_neg[1]}
echo ${w_pos[1]}

index start from 1.

answered May 7, 2020 at 8:33

Md Shihab Uddin

5616 silver badges13 bronze badges

Collectives™ on Stack Overflow

Extracting data from a column in a CSV file with headers and input into an array in Bash

3 Answers 3

Output:

Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Output:

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related