How to remove duplicates from an array of JSON objects in Nodejs

Question

I am trying to remove duplicate entries based on the id for the following json, but it seems I am not able to parse the JSON correctly, My array of JSON is as follows:

[ '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"57"},{"name":"sample","id":"acc"}]}\n',
  '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
  '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
  '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"}]}\n',
]

the output should be

{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"57"},{"name":"sample","id":"acc"}]}

My Code is as follows:

removeDups: async function(listent){
      for (i = 0; i < listent.length; i++){
        console.log(listent.length)
         for (j = 0; j < listent.length; j++){
           console.log(listent[j])
       }
        var standardsList = this.arrUnique(listent);
        console.log("standard", standardsList)
    }
    },

    arrUnique: function (arr) {
      var cleaned = [];
      arr.forEach(function(itm) {
          var unique = true;
          cleaned.forEach(function(itm2) {
              if (_.isEqual(itm, itm2)) unique = false;
          });
          if (unique)  cleaned.push(itm);
      });
      return cleaned;

      }

Thanks. For the second and third items, acc and 57 are the unique ids, but because the first items ids are all the same, the id used will be the id shared by all of them, is that the rule? Find the unique id, or use whatever ID is shared by all? — CertainPerformance
– CertainPerformance, Commented Oct 13, 2018 at 1:50
@NidhiSharma you could use _.uniq to solve it. But, you will need to parse the response into form that can be ingested into JS (using JSON.parse). — trk
– trk, Commented Oct 13, 2018 at 4:14

trk · Accepted Answer · 2018-10-13 07:05:53Z

1

hope this works for you: https://jsfiddle.net/c0pkw5ju/

The core functions are:

var val = array.reduce(function (acc, val) {
    return acc.concat(val.entries)
}, [])

console.log(_.unique(val, function (v) {
    return v.id
}))

Basically, you need to convert your \n separated JSON response to a JS object which can be consumed as indicated. Let me know if this not help.

Output:

[
  {
    "name": "sample",
    "id": "8d"
  },
  {
    "name": "sample",
    "id": "57"
  },
  {
    "name": "sample",
    "id": "acc"
  },
  {
    "name": "sample",
    "id": "50"
  },
  {
    "name": "sample",
    "id": "ac"
  }
]

UPDATE: Assuming response to be:

var response = ['{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"57"},{"name":"sample","id":"acc"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"}]}\n',
]

If you processed response as indicated here:

var array = response.map(function(val) {
    return JSON.parse(val.trim())
})

then core functions should work.

edited Oct 13, 2018 at 7:05

answered Oct 13, 2018 at 4:13

trk

2,23815 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Nidhi Sharma Over a year ago

I tried this and it worked absolutely fine, the only issue is i am not able to convert the /n from the JSON to convert it into JS Object. I tried JSON.parse but it prints the same output as the arr.

trk Over a year ago

@NidhiSharma hold on ... let me see if I can write something up for that as well...

Dženis H. · Accepted Answer · 2018-10-13 07:19:46Z

1

Run the code snippet and you'll see the exact same output as you mentioned in your question. I hope this will be helpful to you (and to others). Happy coding :)

const data = ['{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"57"},{"name":"sample","id":"acc"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"},{"name":"sample","id":"ac"}]}\n',
    '{"entries":[{"name":"sample","id":"8d"},{"name":"sample","id":"50"}]}\n',
];

// Let us first format it correctly 
const formatted = data.map(x => {
    return JSON.parse(x.trim())
});

// Let's loop over our newly created array and push unique values into a new one

let uniqueNames = [];
let uniqueObj = [];

for(let i = 0; i < formatted.length; i++){    
    if(uniqueNames.indexOf(data[i]._id) === -1){
       uniqueObj.push(data[i])
       uniqueNames.push(data[i]._id);
       }        
    }

// Let's take that one step further and display it exactly as you wanted it to be :)
const iterator = uniqueObj.values();

for (const value of iterator) {
   console.log(value);
}

edited Oct 13, 2018 at 7:19

answered Oct 13, 2018 at 7:02

Dženis H.

7,8923 gold badges30 silver badges49 bronze badges

2 Comments

Dženis H. Over a year ago

@NidhiSharma The pleasure is all mine. Please consider an upvote and accept my answer (when you get the chance to) :)

Nidhi Sharma Over a year ago

sure I have upvoted it already, Can you please take a look at stackoverflow.com/questions/52796929/…, do you have any thoughts on this

Collectives™ on Stack Overflow

How to remove duplicates from an array of JSON objects in Nodejs

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related