sed find replace using bash arrays fails

Question

I have three files, two files with 2259 IP addresses in each file. One file with 137772. The script uses sed with bash arrays and a for loop to replace the IPs in an access.log with different IPs. After several hours of running the script fails with this error:

sed: -e expression #1, char 0: no previous regular expression

The number of uniq IP addresses is also short by six IPs.

Here is the script:

#!/bin/bash
_ORGIFS=$IFS
IFS=$'\n'
_alIPs=($(<access.log.IPs)
_fIPs=($(<randomIPs.txt)
for (( _i=1; _i<=2259; _i++ ))
do 
  sed -i "s/${_alIPs[$_i]}/${_fIPs[$_i]}/g" access.log
done
IFS=$_ORGIFS

Wintermute · Accepted Answer · 2015-04-07 09:27:44Z

3

Array indices in bash start at 0. When you say

for (( _i=1; _i<=2259; _i++ ))

you ignore the first entry and go one past the end, at which point

sed -i "s/${_alIPs[$_i]}/${_fIPs[$_i]}/g" access.log

expands to

sed -i "s//something/g" access.log

The // in the s command attempts to reuse the previously used regular expression, which doesn't exist, and so you get the error.

The solution is to use

for (( _i=0; _i<2259; _i++ ))

...although, seriously, I'd spend some time thinking about ways to do those replacements in bulk.

Addendum: Such as

sed -i -f <(paste access.log.IPs randomIPs.txt | awk '{ print "s/" $1 "/" $2 "/g" }') access.log

(assuming I read your intentions correctly)

answered Apr 7, 2015 at 9:27

Wintermute

44.3k5 gold badges84 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

glenn jackman Over a year ago

Or, just use awk instead of sed with a process substitution.

Nikos M. · Accepted Answer · 2015-04-07 09:31:49Z

0

this is taken from my own bash template lib use as needed:

# parse a template and return it
# 1st arg is template tags [array]
# 2nd arg is replacement values for template tags [array]
# 3rd arg is template file path
# usage: TT=$(parseTemplate TAGS[@] REPLACE[@] $MAIL_TEMPLATE); echo $TT;
function parseTemplate()
{
    local _TAG=("${!1}")
    local _REPLACE=("${!2}")
    local _TEMPLATE="${3}"
    local _PATTERN=""
    local i

    if [[ ${#_TAG[@]} > 0 ]]; then
        _PATTERN="-e s#\%\%${_TAG[0]}\%\%#${_REPLACE[0]}#g"
        for (( i = 1 ; i < ${#_TAG[@]} ; i++ ))
        do
            _PATTERN="${_PATTERN}; s#\%\%${_TAG[${i}]}\%\%#${_REPLACE[${i}]}#g"
        done

        local SED=`which sed`
        $SED "${_PATTERN}" < "${_TEMPLATE}"
    else
        local CAT=`which cat`
        $CAT "${_TEMPLATE}"
    fi
}

The template replacement tags have this format: %%TAG%% one can change this, but this is what this function uses

Example template (file or text):

Hello %%NAME%%, your location is %%LOCATION%%


TAGS = ('NAME' 'LOCATION');
REPLACE = ('Nikos' 'GR')
TT=$(parseTemplate TAGS[@] REPLACE[@] $MAIL_TEMPLATE); 
echo $TT;

http://www.thegeekstuff.com/2010/06/bash-array-tutorial/

edited Apr 7, 2015 at 9:31

answered Apr 7, 2015 at 9:23

Nikos M.

8,3934 gold badges41 silver badges45 bronze badges

2 Comments

glenn jackman Over a year ago

SED=`which sed`; $SED ... offers no advantage to just calling command sed ...

Nikos M. Over a year ago

@glennjackman, ok, this is just taking care of some extreme situations where the command is not in the /bin path (in various environments), one can easily remove these extra code

Collectives™ on Stack Overflow

sed find replace using bash arrays fails

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related