linux command for removing duplicate values

Question

I have 2 files file1 which contains

1,2,3,4,5

file2 contains

4,5,6,7,8

output should be in such a way that new file assume it as file3 should contain

1,2,3,4,5,6,7,8,

and not only that if contents in file1 and file2 changes like

file1 new contents

10,11,12,13,14

file2 new contents

13,14,15,16,17,18

after merging file3 should should contain below values

1,2,3,4,5,6,7,8,10,11,12,13,14,15,16,17,18

I have tried several commands like sort, uniq, cat etc but it haven't worked

touch file3; cat file1 file2 file3 | sort | uniq ? You will need to translate the commas to newlines. sed or tr will do that for you. — Andrew
– Andrew, Commented Jul 30, 2014 at 9:18

Maxim Egorushkin · Accepted Answer · 2014-07-30 09:28:45Z

2

Commands like sort and uniq work on lines.

All you have to do is convert commas to newlines, do sort -u or uniq and then convert the newlines back to commas, e.g.

$ cat a
1,2,3,4,5
$ cat b
4,5,6,7,8
$ cat a b | tr ',' '\n' | sort -u | tr '\n' ','
1,2,3,4,5,6,7,8,

answered Jul 30, 2014 at 9:18

Maxim Egorushkin

138k18 gold badges201 silver badges293 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Hi, Thanq for your reply but I am getting errors the output is not in proper format

@VenkatKrishna I gave you an idea and an example. Try using what you know to adapt it to your needs.

Karata · Accepted Answer · 2014-07-30 09:29:58Z

2

If you want merge result to file3:

cat file1 file2 | sed s/,/\\n/g | sort  -u | tr "\\n" "," >> file3

answered Jul 30, 2014 at 9:29

Karata

1,1498 silver badges17 bronze badges