How to delete duplicate lines from string?

Question

How to write a script in javascript which check all lines ("Line1\nLine2\nLine3...") in a string and if there are duplicate lines then just leave one and ignore br tags?

var s = "Hello world\n<BR>\nThis is some text\nThis is some text\n<BR>\nThis is some text"
line1 = "Hello world"
line2 = "<BR>"
line3 = "This is some text"
line4 = "This is some text"
line5 = "<BR>"
line6 = "This is some text"

var result = "Hello world\n<BR>\nThis is some text\n<BR>"
line 1 = "Hello world"
line 2 = "<BR>"
line 3 = "This is some text"
line 4 = "<BR>"

split on \n, loop through the array, ignore , set to a hash, if it is in hash, than remove, after loop join together. — epascarello
– epascarello, Commented Jun 6, 2013 at 23:30

Fatih Bulut · Accepted Answer · 2019-08-16 08:45:51Z

4

I think the shortest solution is this.

myStr
.split("\n")
.filter((item, i, allItems) => {
  return i === allItems.indexOf(item);
})
.join("\n");

answered Aug 16, 2019 at 8:45

Fatih Bulut

2,6851 gold badge18 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Walter Macambira · Accepted Answer · 2013-06-15 01:47:56Z

1

var pieces = s.split("\n"); //This will split your string
var output = []; //Output array

for (var i = 0; i < pieces.length; i++) { //Iterate over input...

   if (pieces[i] == '<BR>' || output.indexOf(pieces[i]) < 0) { //If it is <BR> or not in output, add to output
      output.push(pieces[i]);
   }

}

var newS = output.join("\n"); //Concatenates the string back, you don't have to do this if you want your lines in the array

Here we have the jsFiddle: http://jsfiddle.net/7s88t/

For you knowledge, the indexOf function returns the position where pieces[i] is at output array. If it is not found, it returns -1. That is why I check if it is less than zero.

Hope I have helped.

EDIT

As you requested, to take lower case:

if (pieces[i].toLowerCase() == '<br>' || pieces[i].toLowerCase() == '<br/>' || pieces[i].toLowerCase() == '<br />' || output.indexOf(pieces[i]) < 0) {

edited Jun 15, 2013 at 1:47

answered Jun 6, 2013 at 23:33

Walter Macambira

2,61521 silver badges28 bronze badges

4 Comments

maniootek Over a year ago

does it check in this order: 1&2, 2&3, 3&4 ?? If so, then how to check 1&3 ?

Walter Macambira Over a year ago

could not understand your question. please be clearer. my code goes from the fisrt line to the last one, adding each one to the output if it has never appeared or is a .

maniootek Over a year ago

sorry I was wrong, just forget. It is working very good but it is not capitalize sensitive. Do you know how to change it?

Walter Macambira Over a year ago

if (pieces[i].toLowerCase() == ' ' || pieces[i].toLowerCase() == ' ' || pieces[i].toLowerCase() == ' ' || output.indexOf(pieces[i]) < 0) {

dudewad · Accepted Answer · 2013-06-07 17:24:44Z

0

1) divide your text into an array by line break:

var arr = s.split("\n");

2) Remove all duplicate entries:

var str;
for(var i=0; i<arr.length; i++) {
    str = arr[i];
    //Takes into account all bad forms of BR tags. Note that in your code you are using
    //invalid br tags- they need to be <br /> (self-closing)
    if(inArray(str, arr) && str != "<br>" && str != "<br/>" && str != "<br />"){
        arr.splice(i, 1);
    }
};

function inArray(needle, haystack) {
    var length = haystack.length;
    for(var i = 0; i < length; i++) {
        if(haystack[i] == needle) return true;
    }
    return false;
}

3) Make them back into a string

//Note joining with newline characters will preserve your line outputs :)
var output = arr.join("\n");

This approach is good because it avoids using regex, doesn't even need to consider   tags, and uses native JS meaning you can put it anywhere you want. I didn't test this code, I just wrote it out so it may contain errors. But it should be a good starting point. Cheers!

edited Jun 7, 2013 at 17:24

answered Jun 6, 2013 at 23:37

dudewad

14.1k7 gold badges43 silver badges51 bronze badges

6 Comments

RobG Over a year ago

Before using inArray, you could first test for Array.prototype.indexOf, which would likely be much faster if available. Oh, and you only need to check from the current index to the end, not the entire array.

dudewad Over a year ago

What @RobG said. Mine was a quick once through but he's right, you should add those other things.

Walter Macambira Over a year ago

You are not taking into account as the OP wants.

dudewad Over a year ago

Ah, I thought he meant "leave them alone" when he said ignore them. If you want to "ignore" br tags (as in leave them out) then simply add the check to your if (inArray) statement. I'll make the edit now. If you want to remove ALL br tags then that's a little more involved and I think you can figure it out on your own anyways :P

dudewad Over a year ago

You should also convert your comparison to lower case when performing the checks because otherwise I think you'll fail checks against upper-case br tags...

|

Collectives™ on Stack Overflow

How to delete duplicate lines from string?

3 Answers 3

Comments

4 Comments

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

4 Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related