how to call single perl script to run parallely through loop for different input parameters

Question

I am new in perl. currently i am running a perl scripts which will call another perl script. the second perl script is having 2 input parameters

sample2.pl -itest.txt -ffile1.txt

I have different input parameters for -f like file1,file2,file3...file10.

Now i want to run the second perl script parallel for all the input parameters(file1,file2,file3) currently i m running as -

#!/usr/bin/perl
use warnings;

use strict;

my $fi="output3.txt";--(output3.txt will contain the files file1,file2..file10)
    open (OF, $fi);

foreach(<OF>)
{
system ("perl ucm3.pl -iinput.txt -f$_ ");

print $_;
}

But it is not runnng parallely it is running one after the other. PLease help to run these script parallely. Thanks in advance.

simbabque · Accepted Answer · 2013-01-29 15:36:03Z

7

You need to create a new process and detach it from your main program. You could do this on foot with fork, but you could also do this with Parallel::ForkManager. It will take care of everything for you.

use strict; use warnings;
use Parallel::ForkManager;

my $pm = Parallel::ForkManager->new($MAX_PROCESSES);

open (my $fh, '<', "output3.txt") or die $!;
while (my $data = <$fh>) {
  chomp $data;

  # Forks and returns the pid for the child:
  my $pid = $pm->start and next;

  # we are now in the child process
  print system ("perl ucm3.pl -iinput.txt -f$data ");

  $pm->finish; # Terminates the child process
}

Edit: If you are not yet familiar with Perl, take a look at this manual. It will tell you how to get Parallel::FormManager (and other things) from CPAN.

edited Jan 29, 2013 at 15:36

answered Jan 29, 2013 at 15:00

simbabque

54.4k8 gold badges77 silver badges141 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

DSD Over a year ago

hi..i used the above solution to run the scripts parellely. it working but the performance is not improved that much. actually ucm3.pl is used to scan the huge directory structure and so it is taking much time to run. Can u please suggest some better way to improve the performance.

simbabque Over a year ago

@Dhiraj Please ask a new question for that, as it is kind of unrelated. Add some of the code of ucm3.pl for the scanning, and explain what parts of your directory structure you are scanning. We'll find a solution then.

DSD Over a year ago

Thanks @simbabque. I have raised a new question for this "How to improve perl script performance?"

score 1 · Accepted Answer · 2013-01-29 16:01:47Z

1

In addition to the other good answers given, you should consider a redesign so that you don't use system to start a new instance of Perl with each process.

Starting all of these perls will add overhead (and that is probably something you care about if you are trying to speed things up with parallel processing).

Ideally, you would make ucm3.pl into a module that can be called from your main program.

But for a quick hack solution, you could just wrap the whole file in a subroutine call:

sub ucm3
{
    #a trick to make the sub arguments look like program arguments.
    local @ARGV = @_;

    [ rest of the ucm3.pl file in here. ]

}

Now, in your program, include the file:

require 'ucm3.pl';

And instead of your system call, you can do this:

ucm3("-iinput.txt", "-f$_");

Combine with fork or threads as suggested in the other answers.

Update: since you are using the same "input.txt" file every time, you could probably make further efficiency gains by refactoring the code so that "input.txt" is only processed once. This would be especially true if that file is large.

edited Jan 29, 2013 at 16:01

answered Jan 29, 2013 at 15:46

user1919238

1 Comment

user1919238 Over a year ago

Thanks. Incidentally, it also demonstrates another valid use of local. stackoverflow.com/a/14458072/1919238

Krishnachandra Sharma · Accepted Answer · 2013-01-29 14:59:28Z

0

For parallel processing you can use threads. The documentation can be studied here.

answered Jan 29, 2013 at 14:59

Krishnachandra Sharma

1,3503 gold badges20 silver badges43 bronze badges

Comments

Miguel Prz · Accepted Answer · 2013-01-29 15:10:06Z

0

If you are working on a Unix like system, you will be able to do a system call where you place an & at the end of the system command:

system "perl ucm3.pl -iinput.txt -f$_ &"

answered Jan 29, 2013 at 15:10

Miguel Prz

13.8k32 silver badges42 bronze badges

Comments

Lukas Eklund · Accepted Answer · 2013-01-29 15:47:58Z

You can also do this with threads and Thread::Queue. It's a little more verbose than Parallel::ForkManager but it's easy to manage and can easily be modified to capture all the outputs and pass them off to another queue.

#!/usr/bin/env perl

use warnings;
use strict;

use threads;
use Thread::Queue;
use contant MAX_THREADS => 10;

# create a queue that we will fill with work
my $q = Thread::Queue->new();

open (my $fh, '<', "output3.txt") or die "cannot open output3.txt $!";
while (my $data = <$fh>) {
  chomp $data;
  # add each file to the queue
  $q->enqueue($data);
}

for (1..MAX_THREADS) {
  # create some faux signals to end work
  $q->enqueue("SIGEXIT");
  # create threads and do work
  threads->create("work");
}

# wait until threads are all done
while (threads->list(threads::running)) {
  sleep 1;
}
print "all done\n";

# subroutine each thread performs
sub work {
  while (my $file = $q->dequeue()) {
    last if $file eq 'SIGEXIT';
    print system ("perl ucm3.pl -iinput.txt -f$file");
  }
  # detach thread for automatic cleanup
  threads->detach;
}

Collectives™ on Stack Overflow

how to call single perl script to run parallely through loop for different input parameters

5 Answers 5

3 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related