Remove duplicates values from list of dictionaries in python

Question

I want to remove duplicates values from list which is inside the dictionary. I am trying to make configurable code to work on any field instead of making field specific.

Input Data :

{'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['[email protected]', '[email protected]']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280, 12177218280]}]}

Expected Output Data :

{'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['[email protected]']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280]}]}

code tried:

dic = {'Customer_Number': 90617174, 'Email': [{'Email_Type': 'Primary', 'Email': ['[email protected]', '[email protected]']}], 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280, 12177218280]}]}

res = []
for i in dic:
    if i not in res:
        res.append(i)

What you are trying right now is just adding the dic keys to the res list — Mortz
– Mortz, Commented Nov 22, 2021 at 9:03
@Mortz I am new to python and don't have much idea how to resolve this. I have just put what i tried. — Naveen Gupta
– Naveen Gupta, Commented Nov 22, 2021 at 9:06

mama · Accepted Answer · 2021-11-22 09:20:20Z

1

You can use set()

import json 

dic = {
    'Customer_Number': 90617174,
    'Email': [
        {
            'Email_Type': 'Primary',
            'Email': list(set([
                '[email protected]',
                '[email protected]',
            ]))
        }
    ],
    'Phone_Number': [
        {
            'Phone_Type': 'Mobile',
            'Phone': list(set([
                12177218280,
                12177218280,
            ]))
        }
    ]
}

print(json.dumps(dic,indent=2))

If you want to do it on a list of dic's then you can do like this:

for dic in dics:
    for email in dic['Email']:
        email['Email'] = list(set(email['Email']))
    
    for phone in dic['Phone_Number']:
        phone['Phone'] = list(set(phone['Phone']))

edited Nov 22, 2021 at 9:20

answered Nov 22, 2021 at 9:05

mama

2,2571 gold badge12 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Naveen Gupta Over a year ago

I can't use the set inside the data. I have just given dic for the reference.

Mortz Over a year ago

@mama Are you suggesting editing the data structure itself? I think that would become cumbersome very quickly

mama Over a year ago

Nope I am just using the OP's own data to show him that 2 can become 1 with set() the more general implementation I will leave up to him self :)

Naveen Gupta Over a year ago

@mama, your answer more specific to field. If you have 100 fields and same duplicate issue then it won't be effecitve ans.

Mortz · Accepted Answer · 2021-11-22 09:26:26Z

0

The approach that you started with, you need to go a few levels deeper with that to find every such "repeating" list and dedupe it.

To dedupe, you can use a set - which is also a "container" data structure like a list but with some (many?) differences. You can get a good introduction to all of this in the official python docs -

for key in dic:
    if isinstance(dic[key], list):
        for inner_dict in dic[key]:
            for inner_key in inner_dict:
                if isinstance(inner_dict[inner_key], list):
                    inner_dict[inner_key] = list(set(inner_dict[inner_key]))

print(dic)
#{'Customer_Number': 90617174,
# 'Email': [{'Email_Type': 'Primary', 'Email': ['[email protected]']}],
# 'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280]}]}

answered Nov 22, 2021 at 9:26

Mortz

4,9691 gold badge23 silver badges39 bronze badges

1 Comment

mama Over a year ago

You could make a recursive function or use visitor pattern to deep dive.

Collectives™ on Stack Overflow

Remove duplicates values from list of dictionaries in python

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related