Delete duplicates strings from big array

Question

How can I remove duplicates from 10.000 elements array of strings?

I have array of strings in format:

[ '[[0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0],[0,0,0,0,0,0,0,0,0], 
[0,0,0,1,1,1,1,1,1],[0,0,0,1,1,1,1,1,1],[0,0,0,1,1,1,1,1,1], 
[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0]]',
'[[0,0,0,1,1,1,1,1,1],[0,0,0,1,1,1,1,1,1],[0,0,0,1,1,1,1,1,1], 
[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0], 
[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0],[0,0,0,1,1,1,0,0,0]]', ....]

There are 10.000 of elements in it, and after deleting duplicates it should be about 500. For now i'm using this code, but it doesnt work since "FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory" problem is poping out.

newarr = [];

for(var i = 0; i<arr.length; i++){
  var idx = arr.indexOf(arr[i])
  while (idx != 0) {
    newarr.push(idx);
    idx = arr.indexOf(wycinki[i], idx + 1);
  }    
}

What's the expected output?

31piy
– 31piy

2018-07-25 13:00:46 +00:00
Commented Jul 25, 2018 at 13:00 — 31piy
– 31piy, Commented Jul 25, 2018 at 13:00
@3piy array of Strings without duplicates

Mavematt
– Mavematt

2018-07-25 13:04:05 +00:00
Commented Jul 25, 2018 at 13:04 — Mavematt
– Mavematt, Commented Jul 25, 2018 at 13:04

Aplet123 · Accepted Answer · 2018-07-25 13:04:12Z

5

A one-liner that can do it utilizes the Javascript Set class.

Array.from(new Set(arr));

This code will return an array of all unique strings.

answered Jul 25, 2018 at 13:04

Aplet123

35.8k1 gold badge41 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mario Over a year ago

very elegant solution

Collectives™ on Stack Overflow

Delete duplicates strings from big array

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related