Perl: STDOUT/the output of shell command to an array directly

Question

I have to access a shell command - hive within a Perl script, So I use `...`. Assuming the result of `hive ... ...` contains 100000000 lines and is 20GB size. what I want to achieve is like this:

@array = `hive ... ...`;

Does `` automatically know to use "\n" as separator to divide each line into the @array?

The 2 ways I can thought of are (but with problem in this case):

$temp = `hive ... ...`;
@array = split ( "\n", $temp );
undef $temp;

The problem of this way is that if the output of hive is too big in this case, the $temp cant store the output, resulting in segmentation fault core dump.

OR

`hive ... ... 1>temp.txt`;
open ( FP, <, "temp.txt" );
while (<FP>)
{
    chomp;
    push @array, $_;
}
close FP;
`rm temp.txt`;

But this way would be too slow, because it writes result first to hard-disk.

Is there a way to write the output of a shell command directly to an array without using any 'temporary container'?

Very Thanks for helping.

mob · Accepted Answer · 2015-09-15 01:42:56Z

3

@array = `command`;

does, in fact, put each line of output from command into its own element of @array. There is no need to load the output into a scalar and split it yourself.

But 20GB of output stored in an array (and possibly 2-3 times that amount due to the way that Perl stores data) will still put an awful strain on your system.

The real solution to your problem is to stream the output of your command through an IO handle, and deal with one line at a time without having to load all of the output into memory at once. The way to do that is with Perl's open command:

open my $fh, "-|", "command";
open my $fh, "command |";

The -| filemode or the | appended to the command tells Perl to run an external command, and to make the output of that command available in the filehandle $fh.

Now iterate on the filehandle to receive one line of output at a time.

while (<$fh>) {
    # one line of output is now in $_
    do_something($_);
}
close $fh;

answered Sep 15, 2015 at 1:42

mob

119k18 gold badges159 silver badges291 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

David Young Over a year ago

Thanks so much. I've tried your way. Just concerned about high memory consumption and putting 20gb into ram is still very slow. The way's fantastic, fast and low memory usage.

Collectives™ on Stack Overflow

Perl: STDOUT/the output of shell command to an array directly

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related