bash & sed: replace multiple strings in variable with for loop

Question

Using bash, I'm trying to replace multiple lines in a stored variable with sed and then print the variable with the line changes.

The list variable looks like this - the virtmx elements will all be unique - non repeating

$list variable:

domain1.com     =====>  dest.domain.com:25
domain2.com     =====>  virtmx-0050:25
domain3.com     =====>  dest.domain2.com:25
domain4.com     =====>  domain.example.com:25
domain5.com     =====>  dest.domain3.com:25
domain6.com     =====>  virtmx-0051:25
domain7.com     =====>  dest.domain4.com:25
domain8.com     =====>  dest.domain5.com:25
domain9.com     =====>  dest.domain6.com:25

where I want to replace all virtmx-[num] with another line from a file on the system.

somefile.log has both line[num] and the replacement line together:

example of lines in file.log:

cat /somedir/somefile.log | grep virtmx-*
<mta id="50" host="virtmx-0050:25" dns_type="MX" desc="mxgroupname">
<mta id="51" host="virtmx-0051:25" dns_type="MX" desc="mxgroupname2">

example of line extraction with awk:

cat /somedir/file.log | grep virtmx-0050 | awk -F "\"" '{print $4,$8}'
virtmx-0050:25 mxgroupname
virtmx-0051:25 mxgroupname2

desired output:

domain1.com     =====>  dest.domain.com:25
domain2.com     =====>  mxgroupname
domain3.com     =====>  dest.domain2.com:25
domain4.com     =====>  domain.example.com:25
domain5.com     =====>  dest.domain3.com:25
domain6.com     =====>  mxgroupname2
domain7.com     =====>  dest.domain4.com:25
domain8.com     =====>  dest.domain5.com:25
domain9.com     =====>  dest.domain6.com:25

$listoflines holds just the lines themselves:

line1
line2

I'm using awk to extract the lines then create a sed replacement string - which I use to replace entries in the $output variable:

line1/replacement1

I think I'm close but the below code is only replacing the first instance in the loop then start over and printing the whole variable again with a new replacement, but missing the first. Instead I want to print the whole variable once with all the replacements. The amount of replacements can vary so that's why I'm trying to use a for loop.

for i in $listoflines
do
    replace=`cat /dir/somemfile.log | grep $i | awk -F "\"" '{print $4,$8}' | sed 's/ /\//'`
    echo "$list" | sed 's/'$replace'/'
done

Does somefile.log have a second line with line2? Also why are you trying to split the file on " rather than spaces. Also, you almost definitly need to read your variable line by line here you will always print your $output lines incorrectly twice. — Reinstate Monica Please
– Reinstate Monica Please, Commented Jan 25, 2014 at 13:00

Community · Accepted Answer · 2017-05-23 10:31:41Z

3

I'm far from an awk guru, but a humble attempt based on glenn's answer:

awk '
    NR==FNR {split($0,x,"\""); r[x[4]]=x[8]; next}
    /virtmx/ {$3=r[$3]}
    {print $1,$2,$3}
' file.log <(echo "$list")

output:

domain1.com =====> dest.domain.com:25
domain2.com =====> mxgroupname
domain3.com =====> dest.domain2.com:25
domain4.com =====> domain.example.com:25
domain5.com =====> dest.domain3.com:25
domain6.com =====> mxgroupname2
domain7.com =====> dest.domain4.com:25
domain8.com =====> dest.domain5.com:25
domain9.com =====> dest.domain6.com:25

discussion:

Sending two 'files' to awk, file.log and <(echo "$list").
NR==FNR is used to process only the first file
1. split file.log on " and assign an associative array r with pattern->replacements.
2. process $list as a file, replace using the assoc array if line matches virtmx, and last print the three columns.

Edit: Removed NR!=FNR by using next in the first line

edited May 23, 2017 at 10:31

CommunityBot

11 silver badge

answered Jan 26, 2014 at 12:00

grebneke

4,51419 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

glenn jackman Over a year ago

I would just {print} and if you need to format the output nicely, pipe into | column -t}, but otherwise more or less what I'd write.

grebneke Over a year ago

@glennjackman - thanks. Initially I tried <<<"$list" instead of <(echo "$list") without success. Not sure I understand why - would you mind explaining?

grebneke Over a year ago

@glennjackman - nvm, I think I understand: <<<"$list" sends the contents to awk stdin, but <(echo "$list") acts as a real file using an actual pipe in the filesystem /dev/fd/<n>. Since we're already providing the real file file.log to awk, it will discard stdin but accept /dev/fd/<n>

glenn jackman Over a year ago

precisely: gnu.org/software/bash/manual/bashref.html#Process-Substitution

glenn jackman · Accepted Answer · 2014-01-25 17:45:01Z

0

What you should do:

in awk, instead of creating a sed string "line1/replacement", create an associative array repl[line1] = replacement
then, still in awk, process the $output (use a process substitution to use a file-like thingy), and use if ($NF in repl) $NF = repl[$NF]; print

I'd need to see your existing code to give you more concrete advice.

answered Jan 25, 2014 at 17:45

glenn jackman

249k42 gold badges233 silver badges362 bronze badges

1 Comment

EightBall Over a year ago

I've edited the whole post for clarity. Can you give me an example of the array with my code? I've never worked with arrays and awk before.

Collectives™ on Stack Overflow

bash & sed: replace multiple strings in variable with for loop

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related