scp multiple arguments of bash script

Question

I am trying to use scp to copy multiple files from a remote server.

I can successfully do it for a single argument as script file_number

Where script is:

#!/bin/bash
echo Insert your username:
read user
scp $user@server:/path/run$1*.lz4 ./
lz4 -mdv --rm run$1*.lz4

All my files have the format: runXXXXsubY.lz4

This way I can copy all the files that have XXXX==file_number, but I want to pass multiple arguments, and copy all those files e.g. script 1 2 to copy all the files with xxxx==1 and XXXX==2.

I tried with:

scp $user@server:/path/run"$@"*.lz4 ./

But $@ gets expanded without appending the *.lz4

I appreciate any help in the right direction.

Thanks

It is fraught with peril and quoting issues, but you could do scp $(printf "$user@server:/path/run%s.lz4 " "$@") ./. You're better off with xargs. — William Pursell
– William Pursell, Commented Sep 25, 2021 at 23:17
You're better off with a simple for loop calling scp once for each file. There's not much benefit from calling scp with multiple args over calling it multiple times. — William Pursell
– William Pursell, Commented Sep 25, 2021 at 23:18
@WilliamPursell The problem with the for loop is that it asks for the password as many times as there are arguments. Which is something I want to avoid. — user17004502
– user17004502, Commented Sep 25, 2021 at 23:20
You should set up your ssh keys so that no password is required. — William Pursell
– William Pursell, Commented Sep 25, 2021 at 23:21
If you need to copy a lot of files, I would look at how to run ssh in master mode so that each scp can reuse the same authenticated connection for each transfer, rather than having to reauthenticate for each file. — chepner
– chepner, Commented Sep 26, 2021 at 0:40

Gordon Davisson · Accepted Answer · 2021-09-26 01:30:39Z

2

The basic problem is that when the shell expands $user@server:/path/run"$@"*.lz4, it doesn't copy the $user:... and *.lz4 parts for each argument, it just kind of blindly adds the list of arguments -- including word breaks between arguments -- into the middle. So if the args are 1 and 2, it essentially expands to:

scp $user@server:/path/run"1" "2"*.lz4 ./

...so $user@server:/path/run"1" and "2"*.lz4 are separate arguments to scp, which isn't useful. What you can do is create an array based on the arguments, and then use that as the source list for scp. Something like this:

sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done
scp "${sources[@]}" ./

And then use a separate loop for the lz4 command:

for runNum in "$@"; do
    lz4 -mdv --rm run${runNum}*.lz4
done

EDIT: to avoid having to authenticate multiple times, you can open a master SSH connection and let all the scp transfers piggyback on that. Here's an example, based heavily on Felix Rabe's answer here:

# Create the array of source file patterns to fetch
sources=()
for runNum in "$@"; do
    sources+=("$user@server:/path/run${runNum}*.lz4")
done

# Open master SSH connection:
# (This is the only time you have to enter the password)
sshSocket=~/"$user@server"
ssh -M -f -N -S "$sshSocket" "$user@server"

# Actually copy the files:
scp -o ControlPath="$sshSocket" "${sources[@]}" ./

# Close master connection:
ssh -S "$sshSocket" -O exit "$user@server"

edited Sep 26, 2021 at 1:30

answered Sep 25, 2021 at 23:28

Gordon Davisson

128k16 gold badges141 silver badges168 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user17004502 Over a year ago

This solution still requires me to enter the password for every argument in the script.

Gordon Davisson Over a year ago

@user17004502 I apparently didn't test that part thoroughly. In any case, it can be fixed with a master SSH connection, see update.

William Pursell · Accepted Answer · 2021-09-25 23:54:30Z

1

Your best bet IMO is a simple for loop:

for xxx; do
    scp "$user@server:/path/run${xxx}.lz4" ./ 
done

If you do want to run a single scp (avoiding multiple password prompts is not a motivation for doing this; the way to avoid multiple password prompts is to setup your ssh keys so no password is necessary), you can use xargs:

printf "$user@server:/path/run%s.lz4\0" "$@" | xargs -0 -J % scp % ./

This does not guarantee a single invocation, and xargs will run multiple invocations of scp if enough args are passed that the argument string exceeds certain limits, but this should probably be sufficient.

edited Sep 25, 2021 at 23:54

answered Sep 25, 2021 at 23:20

William Pursell

214k49 gold badges279 silver badges317 bronze badges

Comments

markp-fuso · Accepted Answer · 2021-09-26 00:32:45Z

0

One parameter substitution idea to build a pattern for the remote file names:

$ fn="$@"
$ echo scp "user@server:/path/run{${fn// /,}}*.lz4" ./

Assuming $@ = 222 333 this generates:

scp user@server:/path/run{222,333}*.lz4 ./

NOTE: once satisified with the format remove the echo

edited Sep 26, 2021 at 0:32

answered Sep 25, 2021 at 23:29

markp-fuso

38.6k5 gold badges24 silver badges48 bronze badges

Collectives™ on Stack Overflow

scp multiple arguments of bash script

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related