Remove duplicate objects from JSON file in JavaScript

Question

This is the current JSON file:

[{
    "name": "Peter",
    "age": 30,
    "hair color": "brown"
}, {
    "name": "Steve",
    "age": 55,
    "hair color": "blonde"
}, {
    "name": "Steve",
    "age": 55,
    "hair color": "blonde"
}]

I want to remove the duplicate Steve individual from the list. How can I make a new JSON that checks if the object's name matches and remove any duplicates in JavaScript?

So you only care about the name, and none of the other properties? — cookie monster
– cookie monster, Commented Mar 24, 2014 at 15:58
Order does not matter. And we only care if 'name' matches. If there are two of the same, just have one remaining. — Andrew Kind
– Andrew Kind, Commented Mar 24, 2014 at 16:06
@user2836857 And what if other details in the objects are different? Which one to keep? — thefourtheye
– thefourtheye, Commented Mar 24, 2014 at 16:07
Good question. I am assuming they are all exact but you bring a good point up. — Andrew Kind
– Andrew Kind, Commented Mar 24, 2014 at 16:14

thefourtheye · Accepted Answer · 2014-03-24 16:05:28Z

10

You must load the JSON data in to the program and parse that with JSON.parse, like this

var array = JSON.parse(content.toString())

To filter out the repeated names from the array of Objects, we use Array.prototype.filter function. You can store the names in an object, and next time when the same name appears we simply filter it out from the result.

var seenNames = {};

array = array.filter(function(currentObject) {
    if (currentObject.name in seenNames) {
        return false;
    } else {
        seenNames[currentObject.name] = true;
        return true;
    }
});

console.log(array);
# [ { name: 'Peter', age: 30, 'hair color': 'brown' },
#   { name: 'Steve', age: 55, 'hair color': 'blonde' } ]

edited Mar 24, 2014 at 16:05

answered Mar 24, 2014 at 15:58

thefourtheye

241k53 gold badges466 silver badges505 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Shijo Rs · Accepted Answer · 2018-11-14 06:26:31Z

5

var data = [{
    "name": "Peter",
    "age": 30,
    "hair color": "brown"
}, {
    "name": "Steve",
    "age": 55,
    "hair color": "blonde"
}, {
    "name": "Steve",
    "age": 55,
    "hair color": "blonde"
}]
 data = this.data.filter((obj, pos, arr) => {
            return arr.map(mapObj =>
                  mapObj.name).indexOf(obj.name) == pos;
            });
          console.log(data);

answered Nov 14, 2018 at 6:26

Shijo Rs

1571 silver badge2 bronze badges

Comments

Timothy Shields · Accepted Answer · 2014-03-24 15:57:10Z

2

Using Underscore.js and the uniq function:

_.uniq(array, false, function (item) { return item.name; })

answered Mar 24, 2014 at 15:57

Timothy Shields

80.3k21 gold badges127 silver badges173 bronze badges

5 Comments

thefourtheye Over a year ago

No downvote but I don't think OP mentioned anything about underscore.

Timothy Shields Over a year ago

@thefourtheye I'm well aware. The answer isn't wrong, it just suggests using a JS tool.

cookie monster Over a year ago

It's irrelevant if OP doesn't want to include a whole library just for this one task. You should make that determination first.

Timothy Shields Over a year ago

@cookiemonster I just am not a fan of reinventing the wheel every time a simple problem presents itself. Yes, the code I gave suggests using a library, and certainly people often do not want to add another library to the mix. However, it is extremely declarative code. thefourtheye's answer, for example, is not nearly as concise and immediately understandable. To each his own I guess.

thefourtheye Over a year ago

@TimothyShields Even I like using underscore :)

tymeJV · Accepted Answer · 2014-03-24 15:58:30Z

1

Loop, check, splice, repeat:

var distinctValues = {};
for (var i = 0; i < data.length; i++) {
    if (distinctValues.hasOwnProperty(data[i].name]) {
        //already has it
        data.splice(i, 1);
        i--;
    } else {
        distinctValues[data[i].name] = true;
    }
}

answered Mar 24, 2014 at 15:58

tymeJV

105k14 gold badges165 silver badges158 bronze badges

2 Comments

cookie monster Over a year ago

You'll need to decrement i when doing a .splice(), or iterate in reverse.

tymeJV Over a year ago

@cookiemonster -- Damnit, I always forget that... :\

Frans Sjöström · Accepted Answer · 2022-05-19 12:02:16Z

0

This was the best solution I could find that makes you able to filter on multiple values in you json object. Solution without _ (module)

array.filter((thing, index, self) =>
    index === self.findIndex((t) => (
    t.place === thing.place && t.name === thing.name // you can add more arguments here to filter more
  ))
)

//Other example
array.filter((thing, index, self) =>
    index === self.findIndex((t) => (
    t.place === thing.place && t.name === thing.name && t.time === thing.time
  ))
)

answered May 19, 2022 at 12:02

Frans Sjöström

4654 silver badges8 bronze badges

Comments

Andrew Dunai · Accepted Answer · 2014-03-24 16:08:23Z

-3

Python one-liner from CLI:

cat file_with_duplicates.json | python2 -c 'import sys; import json; sys.stdout.write(json.dumps(reduce(lambda x, y: x + [y] if y not in x else x, json.loads(sys.stdin.read()), [])))' > unique_items.txt

answered Mar 24, 2014 at 16:08

Andrew Dunai

3,11921 silver badges27 bronze badges

Collectives™ on Stack Overflow

Remove duplicate objects from JSON file in JavaScript

6 Answers 6

Comments

Comments

5 Comments

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

Comments

5 Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related