2

I'm writing a script to read from a input file, which contains ~1000 lines of host info. The script ssh to each host, cd to the remote hosts log directory and cat the latest daily log file. Then I redirect the cat log file locally to do some pattern matching and statistics.

The simplified structure of my program is a while loop looks like this:

while read host
do
    ssh -n name@$host "cd TO LOG DIR AND cat THE LATEST LOGFILE" | matchPattern
done << EOA
    $(awk -F, '{print &7}' $FILEIN)
EOA

where matchPattern is a function to match pattern and do statistics.

Right now I got 2 questions for this:

1) How to find the latest daily log file remotely? The latest log file name matches xxxx2012-05-02.log and is newest created, is it possible to do ls remotely and find the file matching the xxxx2012-05-02.log file name?(I can do this locally but get jammed when appending it to ssh command) Another way I could come up with is to do 

cat 'ls -t | head -1'  or
cat $(ls -t | head -1)

However if I append this to ssh, it will list my local newest created file name, can we set this to a remote variable so that cat will find the correct file?

2) As there are nearly 1000 hosts, I'm wondering can I do this in parallel (like to do 20 ssh at a time and do the next 20 after the first 20 finishes), appending & to each ssh seems not suffice to accomplish it.

Any ideas would be greatly appreciated!

Follow up: Hi everyone, I finally find a crappy way do solve the first problem by doing this:

ssh -n name@$host "cd $logDir; cat *$logName" | matchPattern

Where $logName is "today's date.log"(2012-05-02.log). The problem is that I can only use local variables within the double quotes. Since my log file ends with 2012-05-02.log, and there is no other files ends with this suffix, I just do a blindly cat *2012-05-02.log on remote machine and it will cat the desired file for me.

Improve this question
4
  • some versions of xargs may be able to help. use man xargs and compare your available options with advice you find here on S.O. searching for [bash] xargs. Good luck. Commented May 2, 2012 at 17:26
  • I did almost this exact thing in Python with Threading and Paramako. I can quary about 2000 machines in 4 minutes. (all that to say, I think this task maybe easier to do with a better tool chest, I don't think shell is the right tool for this) Commented May 2, 2012 at 19:30
  • @tMC Yes I might be using python as well... However the threshold of the problem is not network connection, it's parsing the large log files that might be over 2G large, anyways I might consider python as a bet as well:) Commented May 2, 2012 at 20:18
  • If parsing the files is so time consuming, do it on the remote node. If each node processes its own files, it will be far faster and easier to parallelize Commented May 3, 2012 at 13:38

3 Answers 3

Reset to default
1

For your first question,

ssh -n name@$host 'cat $(ls -t /path/to/log/dir/*.log | head -n 1)'

should work. Note single quotes around the remote command.

For your second question, wrap all the ssh | matchPattern | analyse stuff into its own function, then iterate over it by

outstanding=0
while read host
do
    sshMatchPatternStuff &
    outstanding=$((outstanding + 1))
    if [ $outstanding -ge 20 ] ; then
        wait
        outstanding=$((outstanding - 1))
    fi
done << EOA
    $(awk -F, '{print &7}' $FILEIN)
EOA
while [ $outstanding -gt 0 ] ; do
    wait
    outstanding=$((outstanding - 1))
done

(I assume you're using bash.)

It may be better to separate the ssh | matchPattern | analyse stuff into its own script, and then use a parallel variant of xargs to call it.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Hi dave4420, the first ssh doesn't work for me - because $(ls -t /path/to/log/dir/*.log | head -n 1) is trying to list the files on my local machine rather than the remote hosts /path/to/log/dir. Trying your second solution, thanks!
Which shell are you using locally? The single quotes should force the $(...) to be sent to the remote shell unchanged. Are you sure you're using single quotes, not double quotes or back quotes?
I get /bin/bash when I do echo $SHELL and yes I'm using single quote... for the second question, I wrote the following fucntion sshMatchPatternStuff() { ssh -n user@$3 "cd $1; cat *$2" | matchPattern } Since it's multithread now, I think I need to pass some parameters to matchPattern(I use global variables before), but it uses a pipe, I can't just append params to matchPattern, any ideas about how to do this? Thanks!
It's not multithreaded now, it's multiprocessing: each invocation of sshMatchPatternStuff is in its own process and gets its own copies of the global variables. So I don't think you need to change anything about this.
Re the '$(...)': it's strange that it doesn't work for you. I tested it and it works for me. I'm mystified.
|
1

for your second question, take a look at parallel distributed shell:

http://sourceforge.net/projects/pdsh/

Improve this answer

2 Comments

Thanks johnshen64, but the script is running on my company's machine that is not allowed to install 3rd party programs
oh, in that case there is not much you can do other than try to run remote script in the background.
0

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:

parallel -j0 --nonall --slf <(awk -F, '{print $7}' servers.txt) 'cd logdir; cat `ls -t | head -1` | grep pattern'

This way you get the matching done on the remote server. If you prefer to transfer the full log file and do the matching locally, simply move the grep outside:

parallel -j0 --nonall --slf <(awk -F, '{print $7}' servers.txt) 'cd logdir; cat `ls -t | head -1`' | grep pattern

You can install GNU Parallel simply by:

wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem

Watch the intro videos for GNU Parallel to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.