Removing duplicate objects (based on multiple keys) from array

Question

Assuming an array of objects as follows:

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
]

A duplicate entry would be if label and color are the same. In this case Objects with id = 1 and id = 5 are duplicates.

How can I filter this array and remove duplicates?

I know solutions where you can filter against one key with something like:

const unique = [... new Set(listOfTags.map(tag => tag.label)]

But what about multiple keys?

As per request in comment, here the desired result:

[
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
]

Nina Scholz · Accepted Answer · 2018-11-29 16:02:04Z

37

You could use a Set in a closure for filtering.

const
    listOfTags = [{ id: 1, label: "Hello", color: "red", sorting: 0 }, { id: 2, label: "World", color: "green", sorting: 1 }, { id: 3, label: "Hello", color: "blue", sorting: 4 }, { id: 4, label: "Sunshine", color: "yellow", sorting: 5 }, { id: 5, label: "Hello", color: "red", sorting: 6 }],
    keys = ['label', 'color'],
    filtered = listOfTags.filter(
        (s => o => 
            (k => !s.has(k) && s.add(k))
            (keys.map(k => o[k]).join('|'))
        )
        (new Set)
    );

console.log(filtered);

.as-console-wrapper { max-height: 100% !important; top: 0; }

answered Nov 29, 2018 at 16:02

Nina Scholz

388k26 gold badges367 silver badges417 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Dave Gööck Over a year ago

Some comments on the code would be great for some developers who are not so into closures. This is a great example where closures are a good fit. For those who are interested in: The first function inside listOfTags.filter is a factory function, which is called immediately with a new empty set s. s will be available until the filtering is done. The second function is the actual filter function. It is called with every object o and returns a boolean. (In this case another closure function makes the actual filter test, with the concatenated fields of the object o as params.)

Nina Scholz Over a year ago

@Alfrex92, s is a closure over new Set and o is just every object of the array.

Nina Scholz Over a year ago

@Alfrex92, k is another closure over the next line keys.map(k => o[k]).join('|') of a joint key of some properties.

JCm Over a year ago

What a life saver! I'll get back to this after my project to understand this one.

Redbeard Over a year ago

should be written with cleaner code, using single characters is very confusing

|

Sebastien C. · Accepted Answer · 2020-10-22 09:48:32Z

37

Late one, but I don't know why nobody suggests something much simpler:

listOfTags.filter((tag, index, array) => array.findIndex(t => t.color == tag.color && t.label == tag.label) == index);

answered Oct 22, 2020 at 9:48

Sebastien C.

2,10917 silver badges21 bronze badges

1 Comment

Salman Arshad Over a year ago

Simpler but inefficient, uses nested loops.

COLBY BROOKS · Accepted Answer · 2018-11-29 16:44:54Z

11

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
]

const unique = [];

listOfTags.map(x => unique.filter(a => a.label == x.label && a.color == x.color).length > 0 ? null : unique.push(x));

console.log(unique);

answered Nov 29, 2018 at 16:44

COLBY BROOKS

3291 silver badge6 bronze badges

Comments

charlietfl · Accepted Answer · 2018-11-29 16:01:46Z

7

One way is create an object (or Map) that uses a combination of the 2 values as keys and current object as value then get the values from that object

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
]

const uniques = Object.values(
  listOfTags.reduce((a, c) => {
    a[c.label + '|' + c.color] = c;
    return a
  }, {}))

console.log(uniques)

answered Nov 29, 2018 at 16:01

charlietfl

172k14 gold badges125 silver badges154 bronze badges

Comments

Evert · Accepted Answer · 2018-11-29 15:56:43Z

4

I would tackle this by putting this into temporary Map with a composite key based on the properties you're interested in. For example:

const foo = new Map();
for(const tag of listOfTags) {
  foo.set(tag.id + '-' tag.color, tag);
}

answered Nov 29, 2018 at 15:56

Evert

101k18 gold badges136 silver badges221 bronze badges

2 Comments

Andy Over a year ago

that was one of my first thoughts too, but I find the string concatenation not very elegant.

Evert Over a year ago

@Andy it's not that strange. This is basically how hashmaps works, which is the appropriate data structure for this type of thing.

Felix K. · Accepted Answer · 2018-11-29 18:00:28Z

Based on the assumption that values can be converted to strings, you can call

distinct(listOfTags, ["label", "color"])

where distinct is:

/**
 * @param {array} arr The array you want to filter for dublicates
 * @param {array<string>} indexedKeys The keys that form the compound key
 *     which is used to filter dublicates
 * @param {boolean} isPrioritizeFormer Set this to true, if you want to remove
 *     dublicates that occur later, false, if you want those to be removed
 *     that occur later.
 */
const distinct = (arr, indexedKeys, isPrioritizeFormer = true) => {
    const lookup = new Map();
    const makeIndex = el => indexedKeys.reduce(
        (index, key) => `${index};;${el[key]}`, ''
    );
    arr.forEach(el => {
        const index = makeIndex(el);
        if (lookup.has(index) && isPrioritizeFormer) {
            return;
        }
        lookup.set(index, el);
    });

    return Array.from(lookup.values());
};

Sidenote: If you use distinct(listOfTags, ["label", "color"], false), it will return:

[
    {id: 1, label: "Hello", color: "red", sorting: 6},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
]

Ankit Arya · Accepted Answer · 2019-02-27 15:47:17Z

4

You can use reduce here to get filtered objects.

listOfTags.reduce((newListOfTags, current) => {
    if (!newListOfTags.some(x => x.label == current.label && x.color == current.color)) {
        newListOfTags.push(current);
    }
    return newListOfTags;
}, []);

answered Feb 27, 2019 at 15:47

Ankit Arya

9007 silver badges11 bronze badges

Comments

soumya mishra · Accepted Answer · 2022-03-03 17:01:22Z

3

const keys = ['label', 'color'],
const mySet = new Set();
const duplicateSet = new Set();
const result = objList.filter((item) => {
  let newItem = keys.map((k) => item[k]).join("-");
  mySet.has(newItem) && duplicateSet.add(newItem);
  return !mySet.has(newItem) && mySet.add(newItem);
});

console.log(duplicateSet, result);

This can be used to filter duplicate and non duplicate

answered Mar 3, 2022 at 17:01

soumya mishra

313 bronze badges

Comments

Kamal · Accepted Answer · 2022-05-10 05:51:25Z

2

We can find the unique value by the below script, we can expand the array using forEach loop and check the value exists on the new array by using some() method and after that create the new array by using push() method.

const arr = [{ id: 1 }, { id: 2 }, { id: 4 }, { id: 1 }, { id: 4 }];
        var newArr =[];
        arr.forEach((item)=>{ 
            if(newArr.some(el => el.id === item.id)===false){
                newArr.push(item);
            }  
        }
        ); 
        console.log(newArr);
       //[{id: 1}, {id: 2}, {id: 4}];

answered May 10, 2022 at 5:51

Kamal

7947 silver badges11 bronze badges

Comments

surendra babu · Accepted Answer · 2020-11-26 13:49:43Z

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
];

let keysList = Object.keys(listOfTags[0]); // Get First index Keys else please add your desired array

let unq_List = [];

keysList.map(keyEle=>{
  if(unq_List.length===0){
      unq_List = [...unqFun(listOfTags,keyEle)];
  }else{
      unq_List = [...unqFun(unq_List,keyEle)];
  }
});

function unqFun(array,key){
    return [...new Map(array.map(o=>[o[key],o])).values()]
}

console.log(unq_List);

Ravindr · Accepted Answer · 2022-09-05 08:03:39Z

1

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
];

 const objRes=listOfTags.filter((v,i,s)=>s.findIndex(v2=>['label','color'].every(k=>v2[k]===v[k]))===i);
    console.log(objRes);

answered Sep 5, 2022 at 8:03

Ravindr

1961 gold badge3 silver badges15 bronze badges

Comments

Nijat Aliyev · Accepted Answer · 2021-11-02 18:19:53Z

Maybe helpful. Extract duplicate items from array then delete all duplicates

// Initial database data
[
    { key: "search", en:"Search" },
    { key: "search", en:"" },
    { key: "alert", en:"Alert" },
    { key: "alert", en:"" },
    { key: "alert", en:"" }
]


// Function called
async function removeDuplicateItems() {
    try {
        // get data from database
        const { data } = (await getList());
        
        // array reduce method for obj.key
        const reduceMethod = data.reduce((x, y) => {
            x[y.key] = ++x[y.key] || 0;
            return x;
        }, {});

        // find duplicate items by key and checked whether "en" attribute also has value
        const duplicateItems = data.filter(obj => !obj.en && reduceMethod[obj.key]);
        console.log('duplicateItems', duplicateItems);

        // remove all dublicate items by id
        duplicateItems.forEach(async (obj) => {
            const deleteResponse = (await deleteItem(obj.id)).data;
            console.log('Deleted item: ', deleteResponse);
        });

    } catch (error) {
        console.log('error', error);
    }
}


// Now database data: 
[
    { key: "search", en:"Search" },
    { key: "alert", en:"Alert" }
]

Salman Arshad · Accepted Answer · 2022-04-20 07:59:27Z

One solution is to iterate the array and use a Map of maps to store the value-value pairs that have been encountered so far.

Looking up duplicates this way should be reasonably fast (compared to nested loops or .filter + .find approach).

Also the values could be any primitive type; they are not stringified or concatenated for comparison (which could lead to incorrect comparison).

const listOfTags = [
    {id: 1, label: "Hello",    color: "red",    sorting: 0},
    {id: 2, label: "World",    color: "green",  sorting: 1},
    {id: 3, label: "Hello",    color: "blue",   sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello",    color: "red",    sorting: 6}
];

let map = new Map();
let result = [];
listOfTags.forEach(function(obj) {
    if (map.has(obj.label) === false) {
        map.set(obj.label, new Map());
    }
    if (map.get(obj.label).has(obj.color) === false) {
        map.get(obj.label).set(obj.color, true);
        result.push(obj)
    }
});
console.log(result);

Hardik Desai · Accepted Answer · 2023-05-23 08:36:56Z

const listOfTags = [
    {id: 1, label: "Hello", color: "red", sorting: 0},
    {id: 2, label: "World", color: "green", sorting: 1},
    {id: 3, label: "Hello", color: "blue", sorting: 4},
    {id: 4, label: "Sunshine", color: "yellow", sorting: 5},
    {id: 5, label: "Hello", color: "red", sorting: 6},
]

const removeDuplicate = listOfTags.reduce((prev, item) => {
  const label = item.label;
  const color = item.color;
  const tempPrev = prev;
  if (tempPrev.length) {
    const isExistInPrevArray = tempPrev.some((i) => i.label ===  label && i.color === color);
    if (isExistInPrevArray == false) {
      return [...tempPrev, ...[item]];
    } else {
      return [...tempPrev, ...[]];
    }
  } else {
    return [...tempPrev, ...[item]];
  }
}, []);

console.log('removeDuplicate',removeDuplicate)

Output

removeDuplicate [
  { id: 1, label: 'Hello', color: 'red', sorting: 0 },
  { id: 2, label: 'World', color: 'green', sorting: 1 },
  { id: 3, label: 'Hello', color: 'blue', sorting: 4 },
  { id: 4, label: 'Sunshine', color: 'yellow', sorting: 5 }
]

Collectives™ on Stack Overflow

Removing duplicate objects (based on multiple keys) from array

14 Answers 14

7 Comments

1 Comment

Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

14 Answers 14

7 Comments

1 Comment

Comments

Comments

2 Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related