5

I have read all the similar questions on this topic, but didn't find a matching question to what I am experiencing. I apologise if this has been answered already.

Inside a bash script I wrote, there is a very simple sed command, which does not seem to be working. There are no errors, and the command works perfectly when run from the command line.

In the output from set -x I can see the sed command executing perfectly.

GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)

Bash script: (toned down for easier comprehension)

#!/bin/bash -x

# This script has the exact same sed command as used on cli

contact='"[email protected]"'

sed -i "/$contact/d" /home/tim/Desktop/file.txt

exit

Shell output:

tim@ubuntu:~/Desktop$ cat file.txt
t,b,[email protected]
tim@ubuntu:~/Desktop$ ./test.sh 
+ contact='"[email protected]"'
+ sed -i '/"[email protected]"/d' /home/tim/Desktop/file.txt
+ exit
tim@ubuntu:~/Desktop$ cat file.txt
t,b,[email protected]
tim@ubuntu:~/Desktop$ sed -i "/"[email protected]"/d" /home/tim/Desktop/file.txt
tim@ubuntu:~/Desktop$ cat file.txt
tim@ubuntu:~/Desktop$

I assume I am missing something very obvious, but I am done staring at it hoping for the answer to jump off the screen and slap me in the face. Please help :-)

Tim

Improve this question
13
  • Why do you have 2 quotes in contact='"[email protected]"'. It should just be: contact='[email protected]' Commented Mar 19, 2015 at 10:28
  • Your commands are not identical: inside the script you enclose the match pattern with single quotes, on the cli with doublequotes. That means, sed called from your bash script matches against the email address enclosed in double quotes, which it isn't in your file. Commented Mar 19, 2015 at 10:32
  • @anubhava Sorry, I should I guessed someone would ask that. In the dumbed down script it seems pointless, but in the actual script there is a base64 decoded string, which is then openssl decrypted and becomes the $contact variable, and is quoted. I would rather not try to remove the quotes since the variable is used in many other functions which would then have to be rewritten. Either way, it works on the command line with the quotes. Commented Mar 19, 2015 at 10:33
  • @collapsar That's what I was thinking, except in the script, there are no single quotes, this seems to happen only in the set -x output. Commented Mar 19, 2015 at 10:34
  • it works on the command line with the single quotes ? Commented Mar 19, 2015 at 10:34

2 Answers 2

Reset to default
9

There are double quotes around the mail address in the $contact script variable that are missing from the command line call:

# case 1 - works
# only the sed pattern delimiters are enclosed in quotes and these quotes will be stripped by the shell. 
sed -i "/"[email protected]"/d" ./file.txt; cat file.txt

# case 2 - fails
# escaping with \ turns dquotes #2,3 from shell-level delimiters to char literals w/o special  semantics.
sed -i "/\"[email protected]\"/d" ./file.txt; cat file.txt

# case 3 - fails
# Single quotes enclose the complete sed pattern spec which comprises double quotes enclosing the mail address
sed -i '/"[email protected]"/d' ./file.txt; cat file.txt

# case 4 - works
sed -i "/[email protected]/d" ./file.txt; cat file.txt

# case 5 - works
sed -i '/[email protected]/d' ./file.txt; cat file.txt

This would explain the different behavior of the script vs. the cli call.

The OP pointed out that he needs the double quotes in the real script. that may be so, however, if these doublequotes aren't present in the file, there will be no match.

A solution would be to preprocess the file ( if necessary, work on a copy ) with sed:

sed -i 's/,/","/g; s/^/"/; s/$/"/' ./file.txt

This command assumes a comma-separated list of items on each line with no item containing double quotes. It will wrap each item in double quotes so they will match against the search pattern in the original script's $contact variable.

Alternative (adapted from this SO answer [that I have not been the author of])

Another option is changing the relevant portion of the script be deriving a second variable from $contact:

contact='"[email protected]"'
c2=$(echo $contact | tr -d '"')

sed -i "/$c2/d"  /home/tim/Desktop/file.txt
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Why are the double quotes stripped by the shell and replaced with single quotes? If I could understand that, I would be very happy :-)
The quoting is a generic mechanism to specify string literals containing characters with special semantics to the command processor - eg. whitespace. Double and single quotes differ (as a rule of thumb) in that double quotes still allow for variable expansion. So in any case, sed does not get to see the outermost pair of quotes. In your command line call, there are 2 outermost pairs and effectively 3 adjacent strings that are implicitly wrapped into a single argument to sed (the pattern expression). I've added case 2, contrast that with cases 1,3.
Right, got it, and thank you for such a clear explanation! I think I will have to use your prepocessor to alter the match criteria. Very helpful :-)
3

This is just an addition to collapsar's answer which already solved the issue.

When we use sed in bash script, bash script acts as a wrapper for sed. This has two purposes

  • sed command can be executed as if it is executed outside the bash.

  • bash script wrapper helps sed to communicate with the outside world, using environment variables.

For example, suppose that the file testfile contains two lines

[email protected]
[email protected]

Now If I would like to write a bash script which helps sed replace the lines that contain [email protected] my script sedscript would be like this :

#!/bin/bash
contact='[email protected]'
sed -i "/$contact/d" $1

Now I would execute the script like below

./sedscript testfile

to remove all the lines containing [email protected].

In fact you can replace $1 with the actual file name. But the important point to note, as mentioned in the previous answer, is that whenever we use a bash variable inside the sed command, always enclose the command in double quotes. Only then bash would replace the variable with corresponding string before passing it to sed.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.