Read From Multiple CSV Files in a loop in shell

Question

I have a requirement of reading some CSV files one by one, CSV files will be named Test1.csv,Test2.csv, etc. The code:

     #!/bin/bash
     IFS=","
     FILES=/CSVFiles/*
     CSVNAME=Test
     n=1
     for f in $FILES 
    while read Column1 Column2 
    do
            echo $Column1
            echo $Column2
    done < "$CSVNAME"$n.csv
    n=$((n+1))

Returns these errors:

./ReadCSV.sh: line 23: syntax error near unexpected token `while'

./ReadCSV.sh: line 23: `while read Column1 Column2

Please specify about how many Testn.csv files there are, and whether the files need to be displayed in any particular order. — agc
– agc, Commented Jun 19, 2017 at 4:38
That is a good point to ask as it happens globbing will not consider file order into account — sjsam
– sjsam, Commented Jun 19, 2017 at 4:44
It seems unlikely that /CSVFiles/ is really in the root directory. Please cd to that directory and show the output of readlink -e Test1.csv. — agc
– agc, Commented Jun 19, 2017 at 4:47
@agc there may be n number of csv files.There are two columns in each csv and need to print value of each column row wise. — nkm
– nkm, Commented Jun 19, 2017 at 4:53
@nkm, I don't know what "row wise" means, please give an example of the output. — agc
– agc, Commented Jun 19, 2017 at 5:10

sjsam · Accepted Answer · 2017-06-19 05:02:12Z

1

You're missing a

do

after

for f in $FILES

But if I understood your intentions correctly, you could improve the example using the code below:

#!/bin/bash
for file in /CSVFiles/Test*.csv # Use file globbing to get the files
do
/*Use a stream editor like awk to print the lines*/
  awk -v FS="," '{
  /* if you want the columns in separate lines */
  print $1;
  print $2;
  }' "$file"
done

Edit 2 : If you need to take the file order into consideration, the code below helps

#!/bin/bash
#Count the total number of files using stat and grep
total=$(stat -f /CSVFiles/Test*.csv | grep -c '^[[:blank:]]*File:')
#User c-style for-loops in bash
for((i=0;i<="$total";i++))
do
awk -v FS="," '{
      /* if you want the columns in separate lines */
      print $1;
      print $2;
      }' "Test${i}.csv" # Feeding the file names in order
done

edited Jun 19, 2017 at 5:02

answered Jun 19, 2017 at 4:04

sjsam

22k6 gold badges62 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

nkm Over a year ago

Test1.csv: No such file or directory error at for loop

sjsam Over a year ago

@nkm : What is the exact path to the folder containing those csv files. Are you using my code exactly as it is?

nkm Over a year ago

corrected path and able to read first csv but second csv printing value of 1st csv itself.

Vamsi Sudhir · Accepted Answer · 2017-06-19 04:19:00Z

1

How about this.

IFS=","
FILES=/CSVFiles/*
cat $FILES|while read Column1 Column2
do
echo $Column1
echo $Column2
done

answered Jun 19, 2017 at 4:19

Vamsi Sudhir

111 bronze badge

Comments

agc · Accepted Answer · 2017-06-19 04:31:53Z

0

The output of the OP script seems to be equivalent to this one-liner:

cut -d, -f1,2 /CSVFiles/Test*.csv | tr , '\n'

answered Jun 19, 2017 at 4:31

agc

8,5342 gold badges33 silver badges53 bronze badges

Collectives™ on Stack Overflow

Read From Multiple CSV Files in a loop in shell

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related