Perl script to log external executables output and error while that still runs

Question

I have a Perl script, which runs an external executable. That executable runs for a while (sometimes seconds, sometimes an hour), can spit out text to both STDOUT and STDERR as well as an exit code, which all are needed. Following code demonstrates first successful external executable run (small bash script with one line - the comment), then with bad exit status (example with gs - ghostscript). I want the external executable give its STDOUT to the Perl script for evaluation, filtering, formatting etc. before it gets logged to a logfile (used for other stuff as well) while the external is still executing. STDERR would also be great to be worked on same way. This script is in stand to log everything from STDOUT, but only after the executable has finished. And the STDERR is logged only directly, without evaluations etc. I have no possibility to install any additional Perl parts, modules etc.

How do I get my Perl script to get each line (STDOUT + STDERR) from the executable while it is spitting it out (not just at the end) as well as its exit code for other purposes?

#!/usr/bin/perl
@array_executable_and_parameters = "/home/username/perl/myexecutable.sh" ; #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
@array_executable_and_parameters2= "gs aaa" ;
my $line;
chdir("/home/username/perl/");
$logFileName = "logfileforsomespecificinput.log";
open(LOGHANDLE, ">>$logFileName" );
open (STDERR, '>>', $logFileName);                  #Prints to logfile directly
#open (STDERR, '>>', <STDOUT>);                 #Prints to own STDOUT (screen or mailfile)

print LOGHANDLE "--------------OK run\n";
open CMD, '-|', @array_executable_and_parameters or die $@;
while (defined($line = <CMD>)) {                    #Logs all at once at end
    print LOGHANDLE "-----\$line=$line-----\n";
}
close CMD;
$returnCode1= $?>>8;
print LOGHANDLE "\$returnCode1=$returnCode1\n";

print LOGHANDLE "--------------BAD run\n";
open CMD2, '-|', @array_executable_and_parameters2 or die $@;
while (defined($line = <CMD2>)) {
    print LOGHANDLE "-----\$line=$line-----\n";
}
close CMD2;
$returnCode2= $?>>8;
print LOGHANDLE "\$returnCode2=$returnCode2\n";

close(LOGHANDLE);

Take 2. After good advice in comments I have tried the IPC::Run. But something still does not work as expected. I seem to be missing how the looping from start (or pump?) to finish works, as well as how to get it to iterate when I do not know what the last output would be - as the examples everywhere mentions. So far I have now the following code, but it does not work line by line. It spits out listing of files in one go, then waits until the external loop is fully finished to print all the X's out. How do I tame it to the initial needs?

#! /usr/bin/perl
use IPC::Run qw( start pump finish );

@array_executable_and_parameters = ();
push(@array_executable_and_parameters,"/home/username/perl/myexecutable.sh"); #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
my $h = start \@array_executable_and_parameters, \undef, \$out, \$err ;
pump $h;# while ($out or $err);
print "1A. \$out: $out\n";
print "1A. \$err: $err\n";
$out = "";
$err = "";
finish $h or die "Command returned:\n\$?=$?\n\$@=$@\nKilled by=".( $? & 0x7F )."\nExit code=".( $? >> 8 )."\n" ;
print "1B. \$out: $out\n";
print "1B. \$err: $err\n";

Hynek -Pichi- Vychodil · Accepted Answer · 2018-01-10 13:01:04Z

6

Look at IPC modules, especially IPC::Cmd, IPC::Run and if not satisfied then IPC::Run3. There is a lot of details you would have to cover and those modules will make your life a lot easier.

answered Jan 10, 2018 at 13:01

Hynek -Pichi- Vychodil

26.2k5 gold badges55 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Sinan Ünür Over a year ago

While I agree that this answer would be made more useful with the inclusion of a simple example, reviewers please note that the links are to the actual documentation of the modules recommended to solve the issue, not to some transient blog post etc. metacpan is much much less likely to change or disappear.

Borodin Over a year ago

@SinanÜnür: There are many reasons to avoid a link-only solution other than the volatility of those links, and this should have been posted as a comment. A full answer should include an explanation of the errant behaviour, as well as a reason why any links could make a difference. Clearly it is best of all if sample code is also provided.

uldics Over a year ago

Yes, looks promising. But ... I can not install, use external code, modules, dependencies etc. Isn't there a simple, manual way to achieve same results?

Hynek -Pichi- Vychodil Over a year ago

@uldics You mean like simpler than reinventing the wheel? If you would know that open my $fh, '-|', ... is internally just bunch of calls to pipe(), fork(), close(), dup(), execl() or execve() and so on, you wouldn't have to ask. But you ask so the best advice is to stick to the recommended modules. It's not a simple task and a lot of opportunities for nasty bugs.

uldics Over a year ago

IPC::Cmd is unable to interleave STDOUT and STDERR, IPC::Run3 probably waits till finished (does not allow interaction with the subprocess), so from your three suggestions I suppose only one able to satisfy my requirements is IPC:Run. How do I get it to perform like I have asked? I see no examples anywhere with other than file output redirection. Can it do something like open-pipe-while in my example?

|

uldics · Accepted Answer · 2018-03-02 07:18:41Z

OK, have got it to work, so far. Might have some issues - not sure about environment variables, like umask or language related or the system load when push is waiting/blocking, or how to replace die with capturing of all variables for status. Nevertheless for my purpose, seems to work well. Will see how it works on a real system.

#! /usr/bin/perl
BEGIN {
    push @INC, '/home/myusername/perl5/lib/perl5';          #Where the modules from Cpan are
}
use IPC::Run qw( start pump finish );

@array_executable_and_parameters = ();
push(@array_executable_and_parameters,"/home/myusername/perl/myexecutable.sh"); #ls -lh ; for i in {1..5}; do echo X; sleep 1; done
my $h = start \@array_executable_and_parameters, \undef, \$out, \$err ;
while (42) {
    pump $h;# while ($out or $err);
    if ($out eq '' and $err eq '') {last;}
    print "1A. \$out: $out\n";
    print "1A. \$err: $err\n";
    $out = "";
    $err = "";
}
finish $h or die "Command returned:\n\$?=$?\n\$@=$@\nKilled by=".( $? & 0x7F )."\nExit code=".( $? >> 8 )."\n" ;
print "1B. \$out: $out\n";
print "1B. \$err: $err\n";

The key was understanding how the blocking of pump works. All the manuals and help places kind of skipped over this part. So a neverending while which jumps out when pump lets go further without output was the key.

Collectives™ on Stack Overflow

Perl script to log external executables output and error while that still runs

2 Answers 2

7 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related