4

This one has me flummoxed.

I'm creating a Bash script which copies files into a series of repos and then adds them for commit. The files sometimes have spaces in the filenames, so they need to be quoted.

I've created a quoted space-separated list of filenames in a variable in Bash: $x. When I run echo $x I get this:

'test 01.sql' 'test 02.sql' 'test_03.sql'

If I manually run the following (in the appropriate directory), I have no problem:

git add 'test 01.sql' 'test 02.sql' 'test_03.sql'

But in my script, if I run:

git add $x or git add "$x" or git add "${x}", I get a fatal pathspec error from Git.

fatal: pathspec ''test 01.sql' 'test 02.sql' 'test_03.sql'' did not match any files

I've tried both single and double quoted strings with no difference.

The example has been simplified. The full version uses absolute paths to the files. 

'/Volumes/HardDrive/Repo/queries/test 01.sql' '/Volumes/HardDrive/Repo/queries/test 02.sql' '/Volumes/HardDrive/Repo/queries/test_03.sql'

It works when echoed from the script and pasted manually into the git add command, but doesn't work when passed from a variable in the script.

Improve this question
11
  • To better understand why bash behaves the way it does, consider reading mywiki.wooledge.org/BashParser. Or, to make an immediately pertinent point: Quotes from variable expansion are always data, not syntax, as expansions occur only after syntax-level parsing is complete. (This is actually mandatory for correctness: Were it not so, it would be impossible to write shell scripts handling untrusted data safely). Commented Dec 2, 2015 at 21:08
  • mywiki.wooledge.org/BashFAQ/050 is also directly relevant. Commented Dec 2, 2015 at 21:10
  • By the way -- where does your "quoted space-separated list of filenames" come from? If you're building it via string manipulation, the code generating this list is all but certain to have subtle bugs that could be exploited with appropriately crafted filenames. (Filenames are allowed to contain literal quote characters, literal newlines -- anything but NULs; thus, a name containing literal ' characters could potentially escape its quoting and run commands, aka '"$(rm -rf .)"', were a naively quoted string eval'd). Commented Dec 2, 2015 at 21:12
  • Also, echo $x doesn't actually behave the way you expect it to, with the value for x in question. Run printf '%q\n' $x to see the list it actually evaluates to. Commented Dec 2, 2015 at 21:17
  • Your security comments are appreciated! This script is being used only on my local machine to tie SQL backup files to the Git repos for the projects that spawned them. The filenames all come from script generated file globs that originate in the SQL backup folder. I think I'm okay for this application. In the PHP world, eval is evil. So I try not to use it in Bash as well. Might be an over generalization. Commented Dec 2, 2015 at 22:09

3 Answers 3

Reset to default
3

Instead of creating a concatenated string, use an array, for example:

arr=('test 01.sql')
arr+=('test 02.sql')
arr+=('test 03.sql')

Then you'll be able to add the files in the Bash array using:

git add "${arr[@]}"
Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

This works perfectly! I thought that git add wouldn't accept an array. I tested this approach before I went down the rabbit hole of converting my filename arrays into quoted strings. Apparently, Bash array syntax tripped me up once again!
No, git doesn't accept an array ;-) Using the writing style in my example, the shell expands the array to quoted argument list. Git received it as if you typed the arguments yourself manually on the command line
Oh. I didn't understand that. I have to admit that I find the Bash array syntax particularly opaque, especially with when to quote or not quote, when to use [@] or not, or [*], when to use $arr or ${arr} or "${arr}" or even just arr. The simplest thing has me heading for Google. I needed to subtract 1 from a variable and assign it to another variable. I ended up with the clearly suboptimal result=$(( orig - 1 )) (with all the white space characters being critically important). It's hard being a Bash newbie! Thank you for the help!
@ChristopherWerby, ...as for ${foo} vs $foo, the braces only matter if you need to disambiguate or parameterize the expansion; they're otherwise identical. $foo vs "$foo", by contrast, matters deeply; the general rule is to always quote expansions unless you have a specific and compelling reason to do otherwise (which string-splits and glob-expands their results). For "${arr[*]}" vs "${arr[@]}", the former creates a single string from your array (by putting the first character of $IFS between each element); the latter keeps the array elements as separate words.
@ChristopherWerby, ...so, absent a specific and compelling reason, always use "${foo[@]}", not ${foo[*]}, and "$@", not $*. shellcheck.net is useful for catching a lot of the "practice X is almost never correct" scenarios.
|
2

A shell-quoted list of names is a very poor choice of formats to use for programmatic (as opposed to human) input.

This seems nonintuitive, but it's true for a reason: When you type in a command in the shell, that command is parsed as code; it's able to contain redirections, command substitutions, and other expansions with potentially dangerous side effects.

To allow data to be safely handled without any risk of evaluation as code, the shell performs parameter expansion only after most other parsing stages (exclusive of string splitting of expansion results and globbing) are complete.

If you were manually generating this input and reviewing it for correctness, you could use eval:

# THIS IS DANGEROUS unless you trust your string to contain no malicious content!
files="'test 01.sql' 'test 02.sql' 'test_03.sql'"
eval "git add -- $files"

However, if you're programmatically generating this list, format it as a NUL-delimited stream, and use xargs:

# generate a list in unambiguous NUL-delimited form
printf '%s\0' "/path/to/file 1" "/path/to/file 2" >file.txt

# use that list to run `git add` for the named files
xargs -0 git add -- <file.txt

...or a NUL-delimited stream can be read into a shell array:

# read that list into an array
files=( )
while IFS= read -r -d '' filename; do files+=( "$filename" ); done <files.txt

# ...and use the array
git add -- "${files[@]}"
Improve this answer

1 Comment

I'm accepting this as the answer because it answers the original question, which was about passing a string. It also points out, as does janos, that the best approach is passing the array. Bash scripting seems like a dark art. It's surprising how hard it is to do seemingly simple things: assign an array to a variable, pass an array into a function, return an array from a function. Other Stack Overflow questions have helped out a lot, but I wonder if I shouldn't be shelling to Ruby or Python and doing hard stuff there?
0

Is it possible to simply do:

git add .

That way you don't need to specifically reference the filenames that may have spaces (but will stage everything)

Improve this answer

1 Comment

Yes. I could also use git add -A. But my script needs to specifically target the files added to the commit so that I don't inadvertently commit other files which have intentionally not been staged.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.