Get specific column from CSV in Bash script

Question

I have a csv with nearly a hundred of column.

I am writing a bash script to get only few of those column.

CSV:

0.132.XXXXX.1456836300,1456836300,1456814505,900,XXXXXX,762,0.1,3699244,363811201,3025019,482119186,3699204,363808801,3025019,482119186,0,0,0,0,3699153,194638692,3025031,301505084,886378,109026314,2812835,33759500,2138713,85552320,3025031,301505084,3699096,363753480,3699096,363753480,3699096,434036304,3699153,194638692,3699153,283418364,0,0,0,0,0,0,0,0,4291150,4970942734,4286035,4817376889,4291126,4970941294,4286035,4817376889,0,0,4898811554,4286035,4286035,4817376889,4286035,4697533512,4286035,4594668672,0,0,0,0,0,0,4291120,4700737486,4286035,4594668672,4291120,4700737486,3405068,4421499690,3399983,4418549966,4286035,4594668672,4291120,4970939408,0,0,0,0,0
0.132.YYYYYY.1456835400,1456835400,1456813605,900,YYYYYYYY,761,0.1,3642774,359185359,3002739,478806690,3642735,359183019,3002739,478806690,0,0,0,0,3642572,192544856,3002731,299528510,887300,109139720,2755332,33069464,2115491,84623440,3002731,299528510,3642629,359128287,3642629,359128287,3642629,428338238,3642572,192544856,3642572,279966584,0,0,0,0,0,0,0,0,4263398,4930723176,4259146,4779228782,4263375,4930721796,4259146,4779228782,0,0,4860152556,4259146,4259146,4779228782,4259126,4660142239,4259126,4557923215,0,0,0,0,0,0,4263381,4662299857,4259126,4557923215,4263381,4662299857,3376436,4384383822,3372181,4382438954,4259126,4557923215,4263369,4930719910,0,0,0,0,0

I did some research and found this:

while IFS="," read $col1 $col2 ...
do
    echo "$col1|$col2"
done < /home/xxx/Desktop/data.csv

But it requires to put ALL the column (92 in my case...) I can't select want I want. So I try another solution:

cat /home/xxx/Desktop/data.csv | while read line
do
        a=[ cut -d',' -f5 $line];    # get the source        
        b=[ cut -d',' -f3 $line];    # get the received time
    echo "$a  .  $b"
done

It cannot work because of the size of the $line. I get the error name too long or something like that. (and it does not look really optimized but anyway) I guess I am close to it. Any help?

choroba · Accepted Answer · 2016-03-09 13:27:28Z

3

You can use just cut:

cut -d, -f3,18 --output-delimiter='|' data.csv

Note this only works if the comma is never escaped or quoted in the CSV, and no value can contain a newline.

To iterate over the lines and populate variables, you can do

cut -d, -f3,18 --output-delimiter=' ' data.csv | while read col3 col18 ; do
    echo "$col3 | $col18"
done

edited Mar 9, 2016 at 13:27

answered Mar 9, 2016 at 12:50

choroba

245k27 gold badges221 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Charly Roch Over a year ago

In fact I want to separate the column into different variables. Your solution is great and I can get each line easily with a for loop (for output in $(cut...) ). But now I don't know how to set my variables from the $output.

choroba Over a year ago

Ask a new question. You can link to the old one for context.

Collectives™ on Stack Overflow

Get specific column from CSV in Bash script

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related