How to script multiple ssh and scp commands to various systems

Question

I would like to script a sequence of commands involving multiple ssh and scp calls. On a daily basis, I find myself manually performing this task:

From LOCAL system, ssh to SYSTEM1
mkdir /tmp/data on SYSTEM1
from SYSTEM1, ssh to SYSTEM2
mkdir /tmp/data on SYSTEM2
from SYSTEM2, SSH to SYSTEM3

scp files from SYSTEM3:/data to SYSTEM2:/tmp/data
exit to SYSTEM2
scp files from SYSTEM2:/data and SYSTEM2:/tmp/data to SYSTEM1:/tmp/data
rm -fr SYSTEM2:/tmp/data
exit to SYSTEM1
scp files from SYSTEM1:/data and SYSTEM1:/tmp/data to LOCAL:/data
rm -fr SYSTEM1:/tmp/data

I do this process at LEAST once a day and it takes approximately 5-10 minutes going between the different systems and then cleaning up afterwards. I would really like to automate this in a bash script but my amateur attempts so far have been unsuccessful. As you might suspect, the systems communication is constrained, meaning LOCAL can only see System1, System2 can only see System1 and System3, system3 can only see system2, etc. You get the idea. What is the best way to do this? Additionally, System1 is a hub for many other systems so SYSTEM2 must be indicated by the user (System3 will always have the same relative IP/hostname compared to any SYSTEM2).

I tried just putting the commands in the proper order in a shell script and then manually typing in the passwords when prompted (which would already be a huge gain in efficiency) but either the method doesn't work or my execution of the script is wrong. Additionally, I would want to have a command line argument for the script that would take a pattern for which 'system2' to connect to, a pattern for the data to copy, and a target location for the data on the local system.

Such as

./grab_data system2 *05-14* ~/grabbed-data

I did some searching and I think my next step would be to have scripts on each system that perform the local tasks, and then execute the scripts via ssh commands from the respective remote system. Is there a better way? What commands should I look at using and what would be the general approach to this automating this sort of nested ssh and scp problem?

I realize my description may be a bit convoluted so please ask for clarification on any area that I did not properly describe.

Thanks.

Another thing to look at is pssh. See also this and this.

Dennis Williamson
– Dennis Williamson

2012-05-15 00:03:13 +00:00
Commented May 15, 2012 at 0:03 — Dennis Williamson
– Dennis Williamson, Commented May 15, 2012 at 0:03

Community · Accepted Answer · 2017-03-20 10:18:20Z

3

You can simplify this process a lot by tunneling ssh connections over other ssh connections (see this previous answer). The way I'd do it is to create an .ssh/config file on the LOCAL system with the following entries:

Host SYSTEM3
        ProxyCommand    ssh -e none SYSTEM2 exec /usr/bin/nc %h %p 2>/dev/null
        HostName        SYSTEM3.full.domain
        User            system3user

Host SYSTEM2
        ProxyCommand    ssh -e none SYSTEM1 exec /usr/bin/nc %h %p 2>/dev/null
        HostName        SYSTEM2.full.domain
        User            system2user

Host SYSTEM1
        HostName        SYSTEM1.full.domain
        User            system1user

(That's assuming both intermediate hosts have netcat installed as /usr/bin/nc -- if not, you may have to find/install some equivalent way of gatewaying stdin&stdout into a TCP session.)

With this set up, you can use scp SYSTEM3:/data /data on LOCAL, and it'll automatically tunnel through SYSTEM1 and SYSTEM2 (and ask for the passwords for the three SYSTEMn's in order -- this can be a little confusing, especially if you mistype one).

edited Mar 20, 2017 at 10:18

CommunityBot

11 silver badge

answered May 15, 2012 at 4:40

Gordon Davisson

128k16 gold badges141 silver badges168 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Raj Over a year ago

This seems, upfront, to be the best suggestion so far. I tried it out and I can get to "System2" but unfortunately I cannot get to "System3" due to a key/pair lockout. This is something implemented on our side by the administrators of the system. You can only SSH to System3 from System2 with a specific username and apparently when I use the method described above, it's rejecting the key/pair from my local system. Is there a way to direct the proxy connection to use the ssh key from System2 when it connects to System3 instead of using my local key?

Gordon Davisson Over a year ago

If I understand the limitation right, all you need to do is copy the id file (something like ~/.ssh/id_rsa or id_dsa) from SYSTEM2 to LOCAL under an alternate name (like ~/.ssh/SYSTEM3_id_rsa), then add IdentityFile ~/.ssh/SYSTEM3_id_rsa to the SYSTEM3 section of the config file on LOCAL.

Todd A. Jacobs · Accepted Answer · 2012-05-15 00:08:11Z

If you're connecting to multiple systems, and especially if you have to forward connections through intermediate hosts, you will want to use public key authentication with ssh-agent forwarding enabled. That way, you only have to authenticate once.

Scripted SSH with agent forwarding may suffice if all you need to do is check the exit status from your remote commands, but if you're going to do anything complex you might be better off using expect or expect-lite to drive the SSH/SCP sessions in a more flexible way. Expect in particular is designed to be a robust replacement for interactive sessions.

If you stick with shell scripting, and your filenames change a lot, you can always create a wrapper around SSH or SCP like so:

# usage: my_ssh [remote_host] [command_line]
# returns: exit status of remote command, or 255 on SSH error
my_ssh () {
    local host="$1"
    shift
    ssh -A "$host" "$@"
}

Between ssh-agent and the wrapper function, you should have a reasonable starting point for your own efforts.

Community · Accepted Answer · 2017-05-23 11:46:21Z

Another way could be to use rsync, which tomatically creates any needed directories and, if you want, removes the copied source files.

In your case, you could work with the commands

home:~$ ssh system1
system1:~$ ssh system2
system2:~$ rsync -aPSHiv system3:/data /tmp/data
system2:~$ exit
system1:~$ rsync -aPSHiv --remove-source-files system2:/tmp/data /tmp/data
system1:~$ rsync -aPSHiv system2:/data /tmp/data
system1:~$ exit
home:~$ rsync -aPSHiv --remove-source-files system1:/tmp/data /tmp/data
home:~$ rsync -aPSHiv system1:/data /data

If you combine this with Gordon's approach, you can even reduce that to

home:~$ rsync -aPSHiv system1:/data/ system2:/data/ system3:/data/ /data/

Note that rsync makes a difference between ...data and ...data/ - the former means the directory and its contents, the latter just the contents. If you mix them up, you might end up with a directory data in another directory data.

Besides, you simplify things if you work with public SSH keys instead of passwords.

Collectives™ on Stack Overflow

How to script multiple ssh and scp commands to various systems

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related