grep command in my script return empty results but works from command line

Question

I am trying to write a bash script that is suppose to find the file that has a specific string in it. The script calls another script that returns strings of the format:

url=title

title is the string I am looking for. title can have values that look like this for example: 'A Soldier Of The Legion'. I am trying to find the file in the /tmp/audiobooksdirectory that contains the title, 'A Soldier Of The Legion'. All the files in /tmp/audiobooks have names that end with AB.yaml.

Here is the script:

#!/bin/sh

get_pairs='/home/me/util/scripts/get-pairs.sh'
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
for i in `$get_pairs`
do
        echo "pair $i"
        url=`echo $i | cut -d= -f1`
        apptitle=`echo $i | cut -d= -f2- | cut -c1-52`
        echo "grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1"
        the_file=$(grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1)
        echo "the_file=$the_file"
        if [ -z $the_file ]
        then
                echo "No hiera file found for $apptitle ... skipping"
                continue
        fi
        appname=`basename $the_file .yaml`
        echo "url is[$url] and apptitle is [$apptitle] appname is [$appname]"
        exit 0
done
IFS=$SAVEIFS

The output the script produces is this:

pair http://www.example.com/product/B06XK9FGYD='A Soldier Of The Legion'
grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1
the_file=
No hiera file found for 'A Soldier Of The Legion' ... skipping
pair http://www.example.com/product/B01GWQI0OS='Art of War Sun Tzu'
grep -l 'Art of War Sun Tzu' /tmp/audiobooks/*AB.yaml | head -1
the_file=
No hiera file found for 'Art of War Sun Tzu' ... skipping
pair http://www.example.com/product/B0717333MM='Bartleby, the Scrivener (version 2)'
grep -l 'Bartleby, the Scrivener (version 2)' /tmp/audiobooks/*AB.yaml | head -1
the_file=/tmp/audiobooks/BartlebyTheScrivener_AMZAD_AB.yaml
url is[http://www.example.com/product/B0717333MM] and apptitle is ['Bartleby, the Scrivener (version 2)'] appname is [BartlebyTheScrivener_AMZAD_AB]

The odd things is that each of the grep commands I echo out work when I run them from the command line ... for example:

$ grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1
/tmp/audiobooks/A_Soldier_of_the_Legion_AB.yaml

And the script works for the title, 'Bartleby, the Scrivener (version 2)'.

IFS=$'\n\b' for assignment of the literal string as opposed to IFS=$(echo -en "\n\b"). There is no need to use command substitution to set IFS. — David C. Rankin
– David C. Rankin, Commented Jul 2, 2017 at 4:33

janos · Accepted Answer · 2017-07-02 04:28:51Z

If this line:

echo "grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1"

Produces output like this:

grep -l 'A Soldier Of The Legion' /tmp/audiobooks/*AB.yaml | head -1

Then that implies that the value of apptitle includes single-quotes.

You can try this to understand what's happening:

value1='A Soldier Of The Legion'
value2="'A Soldier Of The Legion'"
echo "$value1"
echo "$value2"

Outputs:

A Soldier Of The Legion
'A Soldier Of The Legion'

In other words, what the script really executes is this:

grep -l "'A Soldier Of The Legion'" /tmp/audiobooks/*AB.yaml | head -1

Which will only match if the yaml files contain the titles surrounded by single-quotes.

You probably want to strip the single-quotes from apptitle, for example:

apptitle=$(echo $i | cut -d= -f2- | cut -c1-52 | sed -e "s/^'//" -e "s/'$//")

The sed above will strip the single-quote at each end, and leave other single quotes in the middle of the string alone.

Eliah Kagan · Accepted Answer · 2017-07-02 04:29:57Z

$apptitle contains opening and closing single quote (') characters around the title, and your script passes them to grep.

Reading your script, this is easy to miss, because the commands your script tells the user it is running--and that you are running manually--allow the shell to perform quote removal, and thus do not pass those characters to grep. This is to say that your script is not running the same grep commands you want it to run.
If some of your .yaml files have single quotes around the title being searched for, the script will work for those files. But it will not work for the others. (I expect that is the reason it happens to work for one of your titles.)

Judging by your question's early edit history, it appears you might have considered this possibility and attempted to prevent it by changing

the_file=$(grep -l "$apptitle" /tmp/audiobooks/*AB.yaml | head -1)

to instead read:

the_file=$(grep -l $apptitle /tmp/audiobooks/*AB.yaml | head -1)

That won't help. The 's are still there. (I suggest undoing that change, though it is likely okay because of the value you've assigned IFS.)

When a quoting character like ' appears in the result of parameter expansion ($apptitle), the shell does not treat it specially. It does not prevent word splitting, nor is it subject to quote removal.

For example, when IFS has its default value, the output of x="'a b c'"; printf '(%s)' $x is ('a)(b)(c'), not (a b c). This is to say that, when x holds the value 'a b c', an unquoted $x expands to 'a b c', and word splitting turns that into 'a b c'.

In your case, you have changed IFS so splitting only happens on newlines and backspaces. grep matches lines, so you won't get a newline in a title. Assuming titles don't ever contain backspaces, it's fine (though perhaps stylistically confusing) to keep $apptitle unquoted. But this doesn't remove the enclosing ' characters.

Akif · Accepted Answer · 2017-07-02 03:37:26Z

0

I think you need to place . (dot) when executing get-pairs.sh. Preceding dot means to "source" the contents of that file into the current shell.

for i in `. $get_pairs`

edited Jul 2, 2017 at 3:37

answered Jul 2, 2017 at 3:31

Akif

7,0383 gold badges46 silver badges47 bronze badges

3 Comments

Red Cricket Over a year ago

Thank you for your response but I am receiving the output from the get-airs.sh script as expected.

Red Cricket Over a year ago

I tried your suggestion but my output is unchanged. I wonder if I am mishandling the fact the title could and generally does have spaces in it.

Akif Over a year ago

Did you try debugging your code? You can do it by placing set -x after #!/bin/sh. It is actually difficult to understand the produced output of get-pair.sh and other lines.

Collectives™ on Stack Overflow

grep command in my script return empty results but works from command line

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related