Shell script to run Perl script over every directory in a directory

Question

The setup:

A data directory that contains directories for every day of the year. ie data/2014-01-01/ to 2014-12-31. I have a perl script that I run individually inside each date directory.

I am attempting to run a shell script to run from data and go through each directory from 2014-02-15 to 2014-07-20 and run the perl script inside each directory. The perl script takes about 20 seconds to run. This is what I have so far, it will only run on February so far, and doesn't wait for the perl script to finish. I would like it to run on every directory in the range and wait for the perl script inside the loop to finish before relooping.

 #!/bin/bash

 folders=`find 2014-02*`

 for folder in $folders; do 
 cd $folder
 perl C:/Tools/script.pl
 cd ..
 done

Why not add the folder traversing to the perl script rather than a separate shell script? Is each folder's perl script materially different? — quid
– quid, Commented Feb 3, 2015 at 19:55
1 Don't iterate find results like that, see this answer. 2 cd .. won't take you back if you previously cded deeper than one directory. — Biffen
– Biffen, Commented Feb 3, 2015 at 19:56
find data/ -type d -name '2014-0[2-7]-(1[5-9]|20)' -exec C:/Tools/script.pl {} \; The above command will work I believe (did not test), but it seems more reasonable to build a script that can take an arbitrary start and end point and apply whatever logic you want. Since it seems likely you will have to do this again. — Hunter McMillen
– Hunter McMillen, Commented Feb 3, 2015 at 19:56
@Hunter I'll try this. Will this wait for the perl script to finish over each iteration? And yes I'd like to have it implemented in a script to run again later. — john stamos
– john stamos, Commented Feb 3, 2015 at 19:58

Sobrique · Accepted Answer · 2015-02-03 20:48:14Z

3

Why not do it all in perl? It has perfectly good traversal capability with the File::Find built in module.

Encapsulate your 'script' as a subroutine.

#!/usr/bin/perl

use strict;
use warnings;
use File::Find;

sub your_script_sub {
    my ( $dir ) = @_;
    #do something with $dir. At a worst case, you could just run your script.
    #but there's no real reason to do that, as it's perl already. 
}

sub run_script_in_dirs {
   if ( -d $File::Find::name ) { 
        your_script_sub($File::Find::name);
    }
}

find ( \&run_script_in_dirs, "/path/to/your/dir" );

For bonus points - you could use a thread to parallelise your 'run script in directory:

#!/usr/bin/perl
use strict;
use warnings;
use threads;
use Thread::Queue;

my $num_threads = 4;
my $dir_q = Thread::Queue -> new(); 

sub your_script_sub {
   while ( my $dir = $dir_q -> dequeue() ) {
          # do something in $dir;
   }
}

sub find_dirs_to_run_script {
   if ( -d $File::Find::name ) { 
        $dir_q -> enqueue($File::Find::Name);
    }
}

for ( 1..$num_threads ) {
   threads -> create ( \&your_script_sub );
}

find ( \&find_dirs_to_run_script, "/path/to/dirs" );

$dir_q -> end();

foreach my $thr ( threads -> list() ) { $thr -> join() }

answered Feb 3, 2015 at 20:48

Sobrique

53.6k8 gold badges63 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

john stamos Over a year ago

Looks good. It doesn't look like the range of the directories is included in this though right? Also my other perl script uses another program for analysis so it can only be used once at a time.

Sobrique Over a year ago

No. The way perl does it, is with that subroutine, and you can test either $_ or $File::Find::name for matching the pattern you're seeking. Are you sure your other program can only be used one at a time? Is that some licensing issue?

john stamos Over a year ago

Yes it uses another program for analysis that goes through and calculates a ton of data. Can't run to sets of data at once.

Sobrique Over a year ago

shrug. Well, the first example then. But like I say - it's not particularly common that you have a program that can only run one instance at a time. (Aside from resource constraints). This will traverse anything under /path/to/dirs that is a directory.

john stamos Over a year ago

if ( -d $File::Find::name ) { your_script_sub($File::Find::name); } I'll change name to be the 2015-02-15 to 2015-07-20 range?

|

Collectives™ on Stack Overflow

Shell script to run Perl script over every directory in a directory

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related