Parallel shell execution and parameters

Question

I'm monitoring a folder which is receiving log files. For each log file received, I need to send it to a remote server via SCP. SCP transfer is done via transfer.sh script. Since I need to perform a transfer for each file, its probable that a single file may delay other new files. I would like to "create" a new parallel process for each file in my directory.

MONITOR_FOLDER='/repository/'
PATTERN='log_*'

    for log_file in $MONITOR_FOLDER$PATTERN     
        do              
            echo "$(date +%c) monitor() Processing $log_file CDR file..."
            parallel --will-cite -n0 "sh transfer.sh $log_file 1" ::: {1..1}
        done

the $MONITOR_FOLDER$PATTERN can return 0 or more files. When there is more than 1 file, I want to create a parallel process per file. The following command display the correct list.

ls $MONITOR_FOLDER | grep 'log_*'

Question:

1) For each entry use it as param for my shell script and at the same time create a new process without the loop

Why not just run the command in the background by appending & to it? — Barmar
– Barmar, Commented May 19, 2015 at 1:58
Your greatest bottleneck is the network. Opening a parallel connection to the same server won't alleviate that. (In fact, multiple connections will have more overhead.) — jpaugh
– jpaugh, Commented May 19, 2015 at 1:59
However, you could try gzipping the file locally, piping it through ssh, and then ungzip it on the remote end. That may be faster or slower. YMMV. — jpaugh
– jpaugh, Commented May 19, 2015 at 2:00

Ole Tange · Accepted Answer · 2015-05-19 06:14:52Z

I'm monitoring a folder which is receiving log files. For each log file received, I need to send it to a remote server via SCP. SCP transfer is done via transfer.sh script.

That part is easy:

MONITOR_FOLDER='/repository/'
PATTERN='log_*'

parallel -j0 'echo "$(date +%c) monitor() Processing {} CDR file..."; sh transfer.sh {} 1' ::: $MONITOR_FOLDER$PATTERN

Or:

ls $MONITOR_FOLDER | grep 'log_*' | parallel -j0 'echo "$(date +%c) monitor() Processing {} CDR file..."; sh transfer.sh {} 1'

Since I need to perform a transfer for each file, its probable that a single file may delay other new files. I would like to "create" a new parallel process for each file in my directory.

This is also easy if you allow for a file to be copied more than once and to have as many scp's running as there are files. Simply add & to the command:

MONITOR_FOLDER='/repository/'
PATTERN='log_*'

for log_file in $MONITOR_FOLDER$PATTERN       
    do              
        echo "$(date +%c) monitor() Processing $log_file CDR file..."
        sh transfer.sh $log_file 1 &
    done

Now it gets more tricky if:

You at most want 12 scp's running at the same time
You only want to copy a file once

But you can probably use this: http://www.gnu.org/software/parallel/man.html#EXAMPLE:-GNU-Parallel-as-dir-processor

notifywait -q -m -r -e MOVED_TO -e CLOSE_WRITE --format %w%f $MONITOR_FOLDER |\
grep 'log_*' | parallel -j12 'echo "$(date +%c) monitor() Processing {} CDR file..."; sh transfer.sh {} 1'

It will just sit there waiting for a new file to be written. So if you want to stop it, you will have to kill it.

Amit Kumar · Accepted Answer · 2015-05-19 04:07:07Z

0

I think the problem is in your code :

 for log_file in $MONITOR_FOLDER$PATTERN

Please go through the loop process and study how loop works, in your case for

For e.g.

 for i in '1 2 3 4 5'     # it will iterate from 1 to 5

but

  for i in $VAR    # it will iterate over `echo $VAR` means its value

Thus in your case the variable log_file will get first value as /repository/log_* but not its content.

To make your code working you may do like.

 for log_file in `ls $MONITOR_FOLDER$PATTERN`

answered May 19, 2015 at 4:07

Amit Kumar

3131 gold badge5 silver badges14 bronze badges

Collectives™ on Stack Overflow

Parallel shell execution and parameters

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related