1

I've searched around questions with similar issues but haven't found one that quite fits my situation.

Below is a very brief script that demonstrates the problem I'm facing:

#!/bin/bash

includeString="-wholename './public_html/*' -o -wholename './config/*'"
find . \( $includeString \) -type f -mtime -7 -print

Basically, we need to search inside a folder, but only in certain of its subfolders. In my longer script, includeString gets built from an array. For this demo, I kept things simple.

Basically, when I run the script, it doesn't find anything. No errors, but also no hits. If I manually run the find command, it works. If I remove ( $includeString ) it also works, though obviously it doesn't limit itself to the folders I want.

So why would the same command work from the command line but not from the bash script? What is it about passing in $includeString that way that causes it to fail?

3
  • Your example command expands to this: find . ( -wholename './public_html/*' -o -wholename './config/*' ) -type f -mtime -7 -print. Q: Is that what you want? Does it work when you copy/paste verbatim to the command line? Commented May 6, 2015 at 18:18
  • 1
    Why are you using a directory of . and then filtering with -wholename instead of just using find ./public_html ./config ... -type f -mtime 7 -print? Commented May 6, 2015 at 18:21
  • 1
    BTW, the underlying issue here is covered in BashFAQ #50: mywiki.wooledge.org/BashFAQ/050 Commented May 6, 2015 at 20:40

3 Answers 3

2

You're running into an issue with how the shell handles variable expansion. In your script:

includeString="-wholename './public_html/*' -o -wholename './config/*'"
find . \( $includeString \) -type f -mtime -7 -print

This results in find looking for files where -wholename matches the literal string './public_html/*'. That is, a filename that contains single quotes. Since you don't have any whitespace in your paths, the easiest solution here would be to just drop the single quotes:

includeString="-wholename ./public_html/* -o -wholename ./config/*"
find . \( $includeString \) -type f -mtime -7 -print

Unfortunately, you'll probably get bitten by wildcard expansion here (the shell will attempt to expand the wildcards before find sees them).

But as Etan pointed out in his comment, this appears to be needlessly complex; you can simply do:

find ./public_html ./config -type f -mtime -7 -print
Sign up to request clarification or add additional context in comments.

5 Comments

When I remove the single quotes, I get this error: find: paths must precede expression: ./public_html/uploads but when I follow the other approach - just including the paths - it works. I thought I had tried this approach already and it didn't work, but evidently I was wrong. Thanks!
Good hint re single quotes becoming part of the filename, but the issue that @ChrisRoberts just pointed out is that unquoted use of $includeString makes the words in it subject to up-front pathname expansion by the shell, which breaks the find command (if there are at least two matches.)
P.S.: In other words: it's generally impossible to robustly pass multiple arguments via a single string variable.
Right, and this is why trying to deal with multiple levels of quoting is tricky. You would probably solve it with eval, but I think the solution that discards both quoting and wildcards is better.
Arrays are the most robust solution, followed by xargs, if you must remain POSIX-compliant; given the latter, eval can always be avoided (so as to avoid its security issues).
1

If you want to store a list of arguments and expand it later, the correct form to do that with is an array, not a string:

includeArgs=( -wholename './public_html/*' -o -wholename './config/*' )
find . '(' "${includeArgs[@]}" ')' -type f -mtime -7 -print

This is covered in detail in BashFAQ #50.

1 Comment

+𝟣 for concision and the link, but I hope I didn't slave in the kitchen all day for naught.
1

Note: As Etan points out in a comment, the better solution in this case may be to reformulate the find command, but passing multiple arguments via variable(s) is a technique worth exploring in general.

tl;dr:

The problem is not specific to find, but to how the shell parses command lines.

  • Quote characters embedded in variable values are treated as literals: They are neither recognized as argument-boundary delimiters nor are they removed after parsing, so you cannot use a string variable with embedded quoting to pass multiple arguments simply by directly using it as part of a command.

  • To robustly pass multiple arguments stored in a variable,

    • use array variables in shells that support them (bash, ksh, zsh) - see below.
    • otherwise, for POSIX compliance, use xargs - see below.

Robust solutions:

Note: The solutions assume presence of the following script, let's call it echoArgs, which prints the arguments passed to it in diagnostic form:

#!/usr/bin/env bash
for arg; do     # loop over all arguments
  echo "[$arg]" # print each argument enclosed in [] so as to see its boundaries
done

Further, assume that the equivalent of the following command is to be executed:

echoArgs one 'two three' '*' last  # note the *literal* '*' - no globbing

with all arguments but the last passed by variable.

Thus, the expected outcome is:

[one]
[two three]
[*]
[last]
  • Using an array variable (bash, ksh, zsh):
# Assign the arguments to *individual elements* of *array* args.
# The resulting array looks like this: [0]="one" [1]="two three" [2]="*"
args=( one 'two three' '*' )

# Safely pass these arguments - note the need to *double-quote* the array reference:
echoArgs "${args[@]}" last
  • Using xargs - a POSIX-compliant alternative:

POSIX utility xargs, unlike the shell itself, is capable of recognized quoted strings embedded in a string:

# Store the arguments as *single string* with *embedded quoting*.
args="one 'two three' '*'"

# Let *xargs* parse the embedded quoted strings correctly.
# Note the need to double-quote $args.
echo "$args" | xargs -J {} echoArgs {} last

Note that {} is a freely chosen placeholder that allows you to control where in the resulting command line the arguments provided by xargs go.
If all xarg-provided arguments go last, there is no need to use -J at all.

For the sake of completeness: eval can also be used to parse quoted strings embedded in another string, but eval is a security risk: arbitrary commands could end up getting executed; given the safe solutions discussed above, there is no need to use eval.

Finally, Charles Duffy mentions another safe alternative in a comment, which, however, requires more coding: encapsulate the command to invoke in a shell function, pass the variable arguments as separate arguments to the function, then manipulate the all-arguments array $@ inside the function to supplement the fixed arguments (using set), and invoke the command with "$@".


Explanation of the shell's string-handling issues involved:

  • When you assign a string to a variable, embedded quote characters become part of the string:

    var='one "two three" *' 
    
  • $var now literally contains one "two three" *, i.e., the following 4 - instead of the intended 3 - words, separated by a space each:

    • one
    • "two-- " is part of the word itself!
    • three"-- " is part of the word itself!
    • *
  • When you use $var unquoted as part of an argument list, the above breakdown into 4 words is exactly what the shell does initially - a process called word splitting. Note that if you were to double-quote the variable reference ("$var"), the entire string would always become a single argument.

    • Because $var is expanded to its value, one of the so-called parameter expansions, the shell does NOT attempt to recognize embedded quotes inside that value as marking argument boundaries - this only works with quote characters specified literally, as a direct part of the command line (assuming these quote characters aren't themselves quoted).
    • Similarly, only such directly specified quote characters are removed by the shell before passing the enclosed string to the command being invoked - a process called quote removal.
  • However, the shell additionally applies pathname expansion (globbing) to the resulting 4 words, so any of the words that happen to match filenames will expand to the matching filenames.

  • In short: the quote characters in $var's value are neither recognized as argument-boundary delimiters nor are they removed after parsing. Additionally, the words in $var's value are subject to pathname expansion.

  • This means that the only way to pass multiple arguments is to leave them unquoted inside the variable value (and also leave the reference to that variable unquoted), which:

    • won't work with values with embedded spaces or shell metacharacters
    • invariably subjects the values to pathname expansion

4 Comments

I'm personally wary of recommending xargs without GNU extensions -0 or -d (both of which, obviously, prevent the behavior it's intentionally being used for here) given how easy it is to use unsafely by mistake, but there's a lot of great advice and explanation in this answer. I'm curious -- how long were you working on this? Looks like it was as yet unposted when my much-pithier answer went in.
I appreciate the compliment, @CharlesDuffy. I figured xargs was preferable to eval, security-wise (and I only recommended it for when arrays aren't available). I didn't keep track of how long I worked on this, but a good couple of hours, I'd say (which included my own experiments and deepening my own understanding).
One other option for pure-POSIX shells, by the way, is encapsulating code inside a function and overriding or manipulating "$@" therein. Granted, it's only one array (within each function's scope), but that's often enough.
@CharlesDuffy: Manipulating $@ inside a function probably goes beyond the scope of this answer, but I've added it as a footnote (of sorts); thanks.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.