0

I have a config file in multiple machines which has some duplicated strings in specific line.

Option1 value
Option2 value
Option3 value
# Option X value
# commentary lines
... 
AllowList user1@ip1 user1@ip2 user2@ip3 user2@ip4 user1@ip1 user1@ip2 user2@ip3 user2@ip4 ... 
...
Option Z value

In AllowList line, there are duplicate values. How to get rid of them?

I already know how to remove the duplicate values :

grep AllowList myconfig | tr ' ' '\n' | sort | uniq | xargs

But I want to do that in place, leaving other lines intact.

Improve this question

3 Answers 3

Reset to default
2

How about perl, with uniq (or uniqstr) from List::util?

$ perl -MList::Util=uniq -alpe '$_ = join " ", uniq @F if $F[0] eq "AllowList"' myconfig
Option1 value
Option2 value
Option3 value
# Option X value
# commentary lines
...
AllowList user1@ip1 user1@ip2 user2@ip3 user2@ip4 ...
...
Option Z value

You can add -i for in-place operation.

Improve this answer
1

allowlist.awk:

/AllowList/{
  for(i=1;i<=NF;i++){
    #Check if the field is a repeated in the line, print the field if not.
    if(!a[$i]++){
      printf "%s ",$i
    }
  }
  split("",a) #Equivalent to delete(a)
  print ""    #Print a newline
  next
}
1

Execute the Awk script and overwrite original file.

awk -f allowlist.awk myconfig > temporary
mv temporary myconfig
Improve this answer
-1

Pick out the relevant line from the file, split it into lines, sort with a uniqueness attribute, join it all together again

grep '^AllowList' file | tr ' ' '\n' | LC_ALL=C sort -u | xargs

Using your example, this provides the following result

AllowList user1@ip1 user1@ip2 user2@ip3 user2@ip4

It presupposes that the only "word" on the line starting with a capital letter is the keyword AllowList, and it uses this assumption to place it at the beginning of the sorted line.

Improve this answer
7
  • That's what I did actually. I'll edit my question. Commented Sep 20, 2020 at 15:24
  • Please also explain there why it doesn't work for you Commented Sep 20, 2020 at 15:25
  • It does work but the config file will left unchanged. Commented Sep 20, 2020 at 15:28
  • Please see my edit Commented Sep 20, 2020 at 15:47
  • Of course it's unchanged. You take the output into a new file and then replace the original file. This is Computing 101 Commented Sep 20, 2020 at 17:11

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.