Using sed in a for loop with variables and regex

Question

I'm trying to build a script where a portion of it utilizes 'sed' to tag the filename onto the end of each line in that file, then dumps the output to a master list. The part of the script giving me trouble is sed here:

DIR=/var/www/flatuser
FILES=$DIR/*
for f in $FILES
do
    echo "processing $f file...."
    sed -i "s/$/:$f/" $f
    cat $f >> $DIR/master.txt
done

The issue is that the 'sed' statement works fine outside of the for loop, but when I place it in the script, I believe it's having issues interpreting the dollar signs. I've tried nearly every combo of " and ' that I can think of to get it to interpret the variable and it continuously either puts "$f" at the end of each line, or it fails outright.

Thanks for any input!

@chepner's answer is on-point. The other thing to add to it, though, is that you need to double-quote all expansions to make them safe for contents with whitespace (or glob expansions, etc)... so, for instance, cat "$f" >>"$DIR/master.txt. — Charles Duffy
– Charles Duffy, Commented Mar 22, 2013 at 16:03
Well, bigger point -- if you just want the names included in master.txt, you shouldn't be using sed -i on the original files first; doing it that way means you can't run your script more than once without adding more than one copy of the names, which is an unfortunate restriction. — Charles Duffy
– Charles Duffy, Commented Mar 22, 2013 at 16:15
OHHH I see what's wrong!! posting my answer in a second...hint it's NOT the dollar sign! — pepper
– pepper, Commented Mar 22, 2013 at 16:19

chepner · Accepted Answer · 2013-03-22 16:44:35Z

5

You just need to escape the dollar sign:

sed -i "s/\$/:$f/" "$f"

so that the shell passes it literally to sed.

To expand on Charles Duffy's point about quoting variables:

DIR=/var/www/flatuser
for f in "$DIR"/*
do
    echo "processing $f file...."
    sed -i "s/\$/:${f##*/}/" "$f"
    cat "$f" >> "$DIR/master.txt"
done

If any file names contain a space, it's too late to do anything about it if you assign the list of file names to $FILES; you can no longer distinguish between spaces that belong to file names and spaces that separate file names. You could use an array instead, but it's simpler to just put the glob directly in the for loop. Here's how you would use an array:

DIR=/var/www/flatuser
FILES=( "$DIR"/* )
for f in "${FILES[@]}"
do
    echo "processing $f file...."
    sed -i "s/\$/:${f##*/}/" "$f"
    cat "$f" >> "$DIR/master.txt"
done

For versions of sed that don't use -i, here's a way to explicitly handle the temp file needed to simulate in-place editing:

t=$(mktmp sXXXX); sed "s/\$/:$f/" "$f" > "$t"; mv "$t" "$f" && rm "$t"

edited Mar 22, 2013 at 16:44

answered Mar 22, 2013 at 16:00

chepner

538k77 gold badges594 silver badges746 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

SteveHNH Over a year ago

Thanks for the excellent explanation! Though, when using either of the above methods, I'm still getting sed errors. Here's what I'm seeing and maybe this will help shed some light: sed: -e expression #1, char 7: unknown option to s'. the files it's operating on are similar to this: abc-test1 abc-test2. I thought removing the hyphens in the filename would help, but I saw no change after doing that.

chepner Over a year ago

@pepper is right about the file name's path separator conflicting with sed's delimiter. Although you can use a different delimiter, you still need to choose one that doesn't appear in $f, which could be tricky.

chepner Over a year ago

Removing directories from the value of $f when using it with the sed command removes the issue of / as the delimiter, as a simple file name cannot contain a /.

pepper Over a year ago

If you choose a different delimiter, I don't see why this solution would not work.

chepner Over a year ago

The choice of such a delimiter would always depend on the value of $f.

|

Charles Duffy · Accepted Answer · 2013-03-22 17:53:54Z

3

Personally, I'd do this like so:

dir=/var/www/flatuser
for f in "$dir"/*; do
  [[ $f = */master.txt ]] && continue
  while read -r; do printf '%s:%s\n' "$REPLY" "${f##*/}"; done <"$f"
done >/var/www/flatuser/master.txt

It doesn't modify your files in-place the way sed -i does, so it's safe to run more than one time (the sed -i version will add the names to your files in-place every time it runs, so you'll end up with each line having more than one copy of the filename on it).

Also, sed -i isn't specified by POSIX, so not all operating systems will have it.

edited Mar 22, 2013 at 17:53

answered Mar 22, 2013 at 16:11

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

4 Comments

Charles Duffy Over a year ago

Oops. Earlier revision of this didn't actually open $f for input in the loop.

SteveHNH Over a year ago

This answer gets me the closest to what I'm going for. I'm not married to 'sed' for this solution, it's just what I have in my limited "toolbox" ;). The only issue with this one is that it tags the full path onto the end of each line instead of just the filename. Tried fiddling with it and I dont' see where that comes from. suggestion?

chepner Over a year ago

I'd recommend this answer, over trying to find a delimiter for sed's substitution operator that doesn't occur in $f.

SteveHNH Over a year ago

Exactly what I need! Thanks to everyone for all of your suggestions. I've been spinning my gears on this for too long :).

pepper · Accepted Answer · 2013-03-22 16:39:32Z

0

The problem is NOT the dollar sign. It's that the variable $f contains a "/" character, and sed is using that to separate expressions. Try using "@" as the separator.

DIR=/var/www/flatuser
FILES=$DIR/*
for f in $FILES
do
    echo "processing $f file...."
    sed -i s@"$"@:"$f"@ $f
    cat $f >> $DIR/master.txt
done

edited Mar 22, 2013 at 16:39

answered Mar 22, 2013 at 16:24

pepper

2,2175 gold badges27 silver badges32 bronze badges

9 Comments

chepner Over a year ago

Now you do have a quoting issue, though, because $@ is a special shell parameter :)

pepper Over a year ago

sed statement is in quotes, though I'll edit it to move the quotes and be sure this works

chepner Over a year ago

Double-quotes; parameter expansion applies.

SteveHNH Over a year ago

thanks for the suggestion. I tried changing out the separator earlier to no success. @CharlesDuffy's answer gets me the closest to the end result, thought it puts the full path at the end of the line instead of just the filename

chepner Over a year ago

@SteveHNH: I modified Charle's answer to limit the addition to the simple file name, rather than the full path

|

4dgaurav · Accepted Answer · 2014-07-16 08:15:01Z

0

it's old, but maybe it helps someone. Why not basename the file to get rid of leading directory

DIR=/var/www/flatuser
FILES=( "$DIR"/* )
for f in "${FILES[@]}"
do
    echo "processing $f file...."
    b=`basename $f`
    sed -i "s/\$/:${b##*/}/" "$b"
    cat "$f" >> "$DIR/master.txt"
done

not tested ...

edited Jul 16, 2014 at 8:15

4dgaurav

11.5k4 gold badges35 silver badges59 bronze badges

answered Jul 16, 2014 at 8:09

D-Day

1

Collectives™ on Stack Overflow

Using sed in a for loop with variables and regex

4 Answers 4

6 Comments

4 Comments

9 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

6 Comments

4 Comments

9 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related