Remove duplicate value from nested object using javascript

Question

I have a million of contacts which is from different group. For example

    {"GROUP":[{'G1: [C1,C2,C3........C500000]},
             {'G2': [C1,C2,C3........C500000,D1,D2,D3........D500000]}
.....]
    }

G1 has 500k contacts, G2 has 1 million contacts out of it 500k contacts which is already present in G1.

I want to filter above group object based on the condition, "Contact which is already in any group should be checked and remove from respective group".

Expected Result

 {"GROUP":[{'G1: [C1,C2,C3........C500000]},
             {'G2': [D1,D2,D3........D500000]},....]
    }

Here Group and its contact may increase in size.

What is the best way to implement it using lodash javascript?

Ori Drori · Accepted Answer · 2017-03-10 15:05:35Z

1

If you don't know how many items from previous object, will be included in the next, you can use _.difference() with an external array, which will store all existing items:

var object = {
    group: [
        { g1: ['c1', 'c2', 'c3', 'c4', 'c5']},
        { g2: ['c1', 'c4', 'c5', 'd1', 'd2', 'd3', 'd4', 'd5']},
        { g3: ['c1', 'c3', 'c4', 'c5', 'd1', 'd4', 'd5', 'e1', 'e2', 'e3', 'e4', 'e5']}
    ]
};

var previousItems = [];

var group = object.group.map(function(item) {
  return _.mapValues(item, function(values) {
    var newValues = _.difference(values, previousItems);
    previousItems = previousItems.concat(newValues);
    return newValues;
  });
});

console.log(group);

<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js"></script>

If all items in the previous object are included in the next, you can use a simple Array#slice:

var object = {
    group: [
        { g1: ['c1', 'c2', 'c3', 'c4', 'c5']},
        { g2: ['c1', 'c2', 'c3', 'c4', 'c5', 'd1', 'd2', 'd3', 'd4', 'd5']},
        { g3: ['c1', 'c2', 'c3', 'c4', 'c5', 'd1', 'd2', 'd3', 'd4', 'd5', 'e1', 'e2', 'e3', 'e4', 'e5']}
    ]
};

var prevValues = [];

var group = object.group.map(function(item, index) {
  return _.mapValues(item, function(values) {
    var newValues = values.slice(prevValues.length);
    prevValues = values;
    return newValues;
  });
});

console.log(group);

<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js"></script>

answered Mar 10, 2017 at 15:05

Ori Drori

196k32 gold badges243 silver badges233 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

mymotherland Over a year ago

Works well...I have checked few contacts not 500 millions... Hope it will work well for million contact with great performance.

Ori Drori Over a year ago

I'm quite sure that the client can't handle million contacts, and even if it does, the performance won't be great at all. You'll probably have to partition the handling to several webworkers. This kind of stuff should be handled by the server.

mymotherland Over a year ago

Yes doing it in node js using elastic search api with from index and size :)

Sven Liebig · Accepted Answer · 2017-03-10 12:16:12Z

After a long process of thinking, i have this for you:

It automatically goes throught the object / array structure and removes the duplicated strings in the array of g1, g2, g3, ... and so on.

I never used lodash, so here is vanilla js.

var object = {
    group: [
        { g1: ['a', 'b', 'c', 'd']},
        { g2: ['a', 'b', 'c', 'd', 'e', 'f', 'g']},
        { g3: ['f', 'g', 'h', 'i', 'j', 'k', 'l']}
    ]
};

var duplicateArray = [];

for(var key in object) {
  var group = object[key];
  
  for(var x = 0; x < group.length; x++) {
      var subGroup = group[x];
      
      for(var subkey in subGroup) {
          var contentArray = subGroup[subkey];
          
          for(var y = 0; y < contentArray.length; y++) {
              var value = contentArray[y];
          
              if(duplicateArray.indexOf(value) == -1) {
                  duplicateArray.push(value);
              } else {
                  object[key][x][subkey].splice(y, 1);
                  y -= 1;
              }
          }
      }
  }
}

console.log(`dub ${duplicateArray}`);
console.log(object);

Collectives™ on Stack Overflow

Remove duplicate value from nested object using javascript

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related