Merge multiple csv files using Perl

Question

I am new to Perl so sorry for the simple question.

I have a Perl script that produces 7 csv files. All these files have 8 common column headings.

The file names will be constant, each file only has 8 columns and there will always be data in each value for each column.

The file size for each is never bigger than 400k.

I would like to combine these csv files into one file using Perl. The output will have the same column heading and data from all the 7 files.

So you just want to append the files, but remove headers in between? — amon
– amon, Commented Aug 1, 2013 at 12:26
Why don't you just edit your main Perl script to print to one file instead of 7 different? — TLP
– TLP, Commented Aug 1, 2013 at 12:40
TLP - I am picking up a script off someone who no longer works with me. The system I am pulling the data from only allows one report extraction via odbc per connection, so I have looped the script to produce the individual files. These individual files are required by other users but I also require a combined file. — user2641999
– user2641999, Commented Aug 1, 2013 at 12:59
Do you also want to remove duplicates, if the same row occurs in more than one file? — choroba
– choroba, Commented Aug 1, 2013 at 13:13

amon · Accepted Answer · 2013-08-01 13:16:08Z

6

If you are on some kind of Unix, you can use tail.

$ tail -qn +2 fileA fileB ...

The -q supresses filenames in the output; the -n +2 starts output with the 2nd line.

To get the header as well:

$ (head -n 1 fileA; tail -qn +2 fileA fileB ...) > output-file

If you need to use Perl:

use strict; use warnings; use autodie;
my $files = $#ARGV; # get number of files - 1
while (my $file = shift @ARGV) {
  open my $fh, "<", $file;
  <$fh> unless $files == @ARGV; # discard header unless first file
  print while <$fh>; # output the rest
}

Then: $ perl the-script.pl fileA fileB ...

answered Aug 1, 2013 at 13:16

amon

57.8k2 gold badges93 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Merge multiple csv files using Perl

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related