How to remove duplicates by key-value in javascript object list?

Question

Consider the following array:

[
  { url: "https://url.com/file1", md5: "fbbbabcc19264ce7b376ce4c726b9b85" },
  { url: "https://url.com/file2", md5: "d920d140432b961f07695ec34bd2a8ad" },
  { url: "https://url.com/file3", md5: "fbbbabcc19264ce7b376ce4c726b9b85" },
  { url: "https://url.com/file4", md5: "bf80655dbe90123324f88a778efa39f7" },
  { url: "https://url.com/file5", md5: "fbbbabcc19264ce7b376ce4c726b9b85" }
];

The files "file1", "file3", and "file5" have the same content and therefore the same md5. I only want to keep the files with different md5s (file1, file2, file4).

What are the possible ways to achieve this with modern ES6?

There's no JSON in your question. That's an array of objects. — Andreas
– Andreas, Commented Jan 31, 2020 at 11:43

Nicolae Maties · Accepted Answer · 2020-01-31 11:32:56Z

2

const arr = [
        {url: 'https://url.com/file1', md5: 'fbbbabcc19264ce7b376ce4c726b9b85'},
        {url: 'https://url.com/file2', md5: 'd920d140432b961f07695ec34bd2a8ad'},
        {url: 'https://url.com/file3', md5: 'fbbbabcc19264ce7b376ce4c726b9b85'},
        {url: 'https://url.com/file4', md5: 'bf80655dbe90123324f88a778efa39f7'},
        {url: 'https://url.com/file5', md5: 'fbbbabcc19264ce7b376ce4c726b9b85'}
];


const removeDuplicates = (myArr, prop) => {
    return myArr.filter((obj, pos, arr) => {
        return arr.map(mapObj => mapObj[prop]).indexOf(obj[prop]) === pos;
    });
}
const result = removeDuplicates(arr, 'md5');
console.log(result)

You can filter your array by a certain key, in your case 'md5'.

answered Jan 31, 2020 at 11:32

Nicolae Maties

2,6782 gold badges18 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

maazadeeb · Accepted Answer · 2020-01-31 11:47:34Z

2

You could use a Set to keep track of already found files.

const arr = [{
    url: 'https://url.com/file1',
    md5: 'fbbbabcc19264ce7b376ce4c726b9b85'
  },
  {
    url: 'https://url.com/file2',
    md5: 'd920d140432b961f07695ec34bd2a8ad'
  },
  {
    url: 'https://url.com/file3',
    md5: 'fbbbabcc19264ce7b376ce4c726b9b85'
  },
  {
    url: 'https://url.com/file4',
    md5: 'bf80655dbe90123324f88a778efa39f7'
  },
  {
    url: 'https://url.com/file5',
    md5: 'fbbbabcc19264ce7b376ce4c726b9b85'
  }
]

const uniqueFiles = new Set();
const ans = arr.filter(ele => {
  if (!uniqueFiles.has(ele.md5)) {
    uniqueFiles.add(ele.md5);
    return true;
  }
  return false;
});

console.log(ans)

edited Jan 31, 2020 at 11:47

answered Jan 31, 2020 at 11:40

maazadeeb

6,1122 gold badges32 silver badges41 bronze badges

Comments

Cygra · Accepted Answer · 2020-01-31 11:51:36Z

1

You can use reduce to solve the problem:

const list = [
    { url: 'https://url.com/file1', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
    { url: 'https://url.com/file2', md5: 'd920d140432b961f07695ec34bd2a8ad' },
    { url: 'https://url.com/file3', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
    { url: 'https://url.com/file4', md5: 'bf80655dbe90123324f88a778efa39f7' },
    { url: 'https://url.com/file5', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
];

const removeDup = (arr, key) => {
    return arr.reduce((acc, cur) => {
        if (acc.some(a => a[key] === cur[key])) return acc;
        return acc.concat(cur)
    }, [])
};

console.log(removeDup(list, 'md5'))

answered Jan 31, 2020 at 11:51

Cygra

765 bronze badges

Comments

customcommander · Accepted Answer · 2020-01-31 11:53:57Z

0

Just use a map keyed by md5 hashes:

const md5_map =
  files =>
    files.reduce((map, {url, md5}) =>
      ({...map, [md5]: url}), {});
      
console.log(md5_map(files));

<script>
const files = [
  { url: "https://url.com/file1", md5: "fbbbabcc19264ce7b376ce4c726b9b85" },
  { url: "https://url.com/file2", md5: "d920d140432b961f07695ec34bd2a8ad" },
  { url: "https://url.com/file3", md5: "fbbbabcc19264ce7b376ce4c726b9b85" },
  { url: "https://url.com/file4", md5: "bf80655dbe90123324f88a778efa39f7" },
  { url: "https://url.com/file5", md5: "fbbbabcc19264ce7b376ce4c726b9b85" }
]
</script>

answered Jan 31, 2020 at 11:53

customcommander

19.3k6 gold badges70 silver badges96 bronze badges

2 Comments

Latz Over a year ago

This changes the format of the array: ` { fbbbabcc19264ce7b376ce4c726b9b85: 'url.com/file5', d920d140432b961f07695ec34bd2a8ad: 'url.com/file2', bf80655dbe90123324f88a778efa39f7: 'url.com/file4' } ` ... but it does it job anyway :-)

customcommander Over a year ago

@Latz Yep it's intentional ;)

jdgower · Accepted Answer · 2020-01-31 15:24:26Z

const list = [
  { url: 'https://url.com/file1', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
  { url: 'https://url.com/file2', md5: 'd920d140432b961f07695ec34bd2a8ad' },
  { url: 'https://url.com/file3', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
  { url: 'https://url.com/file4', md5: 'bf80655dbe90123324f88a778efa39f7' },
  { url: 'https://url.com/file5', md5: 'fbbbabcc19264ce7b376ce4c726b9b85' },
];

const deDupe = (arr = []) => {
  return Object.values(arr.reduce((obj, item) => {
    obj[item.md5] = obj[item.md5] || item;
    return obj;
  }, {}));
}

const dedupedList = deDupe(list);

console.log(dedupedList);

/* 
  [
    {
      "url": "https://url.com/file1",
      "md5": "fbbbabcc19264ce7b376ce4c726b9b85"
    },
    {
      "url": "https://url.com/file2",
      "md5": "d920d140432b961f07695ec34bd2a8ad"
    },
    {
      "url": "https://url.com/file4",
      "md5": "bf80655dbe90123324f88a778efa39f7"
    }
  ] 
*/

Collectives™ on Stack Overflow

How to remove duplicates by key-value in javascript object list?

5 Answers 5

Comments

Comments

Comments

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related