Concatenate multiple files listed in a text file using perl Script

Question

I am new at scripting and I am trying to concatenate multiple files, whose paths are listed a text file and output a combined gzip file. for example list file - File_list.txt contains these file paths

/data/path/file1.txt
data2/path2/file2.txt
....file3.txt
....file4.txt

So far my code is for all files listed in a local directory ( outputs only combined file not gzipped):

#!/usr/bin/perl

use strict;
use File::Slurp;

my $directory = 'Users/xyz/Documents/';

opendir(dir, $directory) or die $!;
my @files = readdir(dir);
closedir dir;

my $outfilename = 'Combined.fastq'

my $outfilesrc = undef;

foreach (sort @files){ 
  $outfilesrc.= File::Slurp::slurp("$basedir/$_");
}

open(OUT, "> $basedir/$outfilename") or die ("Can't open for writing: 
  $basedir/$outfilename : $!");
print OUT $outfilesrc;
close OUT;

exit;

Can someone please share how to read the files using this list rather than one single directory? I know it's much easier in simple bash but i am trying to create a module for a pipeline so need this in Perl. Thanks!

Try to read the list into an array, @files. See The correct way to read a data file into an array — Håkon Hægland
– Håkon Hægland, Commented Nov 30, 2015 at 21:50
"I know it's much easier in simple bash" That's only true when "simple bash" is the only language you know well — Borodin
– Borodin, Commented Nov 30, 2015 at 23:40
@Borodin Actually you don't need to "know well" bash scripting: cat `cat file_list` >> new_file is simpler than that perl script — ChatterOne
– ChatterOne, Commented Dec 1, 2015 at 8:29
Well, unless you want to deal with edge cases properly, because that example will break on e.g. filenames with spaces. — Sobrique
– Sobrique, Commented Dec 1, 2015 at 11:27
yes the command line/bash was easy for quick tasks with limited number of files. Since now the files can be scattered all over directory and and from different sources I need to code it to avoid any possible error — AnkP
– AnkP, Commented Dec 1, 2015 at 15:25

Sobrique · Accepted Answer · 2015-11-30 21:55:12Z

3

You don't seem to do anything with a zip file. I can't even begin to guess (Archive::Zip is pretty good though).

For concatenating a bunch of files, you can make use of the ARGV or <> filehandle.

#!/usr/bin/env perl
use strict;
use warnings;

open ( my $combined, '>', 'combined.fastq') or die $!; 

select $combined; 
print while <>; 

close $combined;

Should do the trick - you open an output file, select it as the default place to print, and then print every line captured in <> - which is all the data in any files specified on the command line, or piped data.

So invoking this script as merge.pl *.txt will take all the text files (in the current directory) and merge them into the combined file.

As you've got an input list - that's as simple as:

open ( my $list_of_files, '<', 'file_list.txt' ) or die;
chomp ( @ARGV = <$list_of_files>);
close ( $list_of_files );

This will accomplish the same result overall.

answered Nov 30, 2015 at 21:55

Sobrique

53.6k8 gold badges63 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

AnkP Over a year ago

Thanks a lot for the reply. Yes didn't know I would have to apply this script on bunch of large set files and then zip them up in the end. Also, I might have to use this script on zip files too, so I am thinking of using zcat for that. Would it be the recommended way?

bolav Over a year ago

zcat is for gzipped files. You cannot use zcat on zipped files. Archive::Zip is not for gzipped files. Look at IO::Compress::Gzip and IO::Uncompress::Gunzip for gzip. Look for instance here: unix.stackexchange.com/questions/48690/…

AnkP Over a year ago

Sorry its my fault, I am using it for multiple gzipped files which would be in GBs and since these are huge files I need to keep the scape/time used in mind too.

Sobrique Over a year ago

Different question that. Worth asking separately - and explaining what you are trying to accomplish.

AnkP · Accepted Answer · 2015-12-03 16:36:59Z

0

Thanks a lot for your replies - The script runs well now, being new at perl it sounded difficult one to me. Just posting my code below -

#!/usr/bin/perl
use strict;
use warnings;
use File::Slurp;
use IO::Compress::Gzip qw(gzip $GzipError);


my @data = read_file('./File_list.txt');
my $out = "./test.txt";


foreach my $data_file (@data)

{
    chomp($data_file);
    system("cat $data_file >> $out");
}
my $outzip = "./test.gz";
gzip $out => $outzip;

answered Dec 3, 2015 at 16:36

AnkP

6512 gold badges10 silver badges18 bronze badges

Collectives™ on Stack Overflow

Concatenate multiple files listed in a text file using perl Script

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related