Appending multiple csv files in to a single csv file using shell scripts

Question

I am trying to append multiple csv files in to one. How can I enhance the below script so that an additional column is added. Let's call it "tag". The values in the tag should be the filename from which the record has been appended.

flag=0
for f in $@/*.csv;

do
k=$(wc -l<"$f" )
if [ $flag -eq 0 ];
then
  head -n $k "$f" > out.csv
  flag=1
else

  tail -n +2 "$f" >> out.csv

fi

done

Using @Shawn's approach below I am getting this:-

        $ cat TEST1/a.csv 
    h1,h2,h3 
    a,b,c 
    d,e,f 
    $ cat TEST1/b.csv 
    h1,h2,h3 
    1,2,3 
    4,5,6 
    $ awk 'NR == 1 { print $0 ",tag"; next } 
FNR == 1 { next } 
{ print $0 "," FILENAME }' TEST1/a.csv TEST1/b.csv 
    ,tag2,h3 
    ,TEST1/a.csv 
    ,TEST1/a.csv 
    ,TEST1/b.csv 
    ,TEST1/b.csv

Shawn · Accepted Answer · 2019-03-16 06:40:03Z

1

Something like this using awk:

$ cat a.csv  
header1,header2,header3
a,b,c
d,e,f
$ cat b.csv
header1,header2,header3
1,2,3
4,5,6
$ awk 'NR == 1 { print $0 ",tag"; next }
       FNR == 1 { next }
       { print $0 "," FILENAME }' a.csv b.csv
header1,header2,header3,tag
a,b,c,a.csv
d,e,f,a.csv
1,2,3,b.csv
4,5,6,b.csv

This: Treats the first line of the first file as a header line to print out, skips the first lines of all further files, and prints the remaining lines of all files, appending a column with the current filename to each one.

answered Mar 16, 2019 at 6:40

Shawn

53.9k3 gold badges29 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

itthrill Over a year ago

How can we use the above command in a loop so that it loops through all files within a folder?

Shawn Over a year ago

@itthrill Why would you use a loop beyond the implicit ones in awk? awk '...' *.csv

Shawn Over a year ago

And of course normal redirection of standard output if you want the results in a file.

Collectives™ on Stack Overflow

Appending multiple csv files in to a single csv file using shell scripts

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related