Remove duplicates dictionaries from list of dictionaries in python

Question

I want to remove duplicates dictionaries from list of dictionaries. I am trying to make configurable code to work on any field instead of making field specific.

Input Data

dct = {'Customer_Number': 90617174, 
       'Phone_Number': [{'Phone_Type': 'Mobile', 'Phone': [12177218280.0]}, 
                        {'Phone_Type': 'Mobile', 'Phone': [12177218280.0]}],
       'Email': [{'Email_Type': 'Primary', 
                  'Email': ['[email protected]']},
                 {'Email_Type': 'Primary', 
                  'Email': ['[email protected]']}]
      }

Expected Output:

{'Customer_Number': 90617174, 
                'Email': [{'Email_Type': 'Primary', 
                           'Email': ['[email protected]']}], 
                'Phone_Number': [{'Phone_Type': 'Mobile', 
                                  'Phone': [12177218280]}]}



**Code tried:**

    res_list = []
    for key,value in address_dic.items():
        if isinstance(value,list):
            for i in range(len(value)):
                if value[i] not in value[i + 1:]:
                   res_list.append(value[i])
    
    dic.append((res_list))


**Output Getting**

    type: [[{'Email_Type': 'Primary', 'Email': ['[email protected]']}, {'Email_Type': 'alternate', 'Email': ['[email protected]', '[email protected]']}], [], [{'Phone_Type': 'Mobile', 'Phone': [12177218280.0]}, {'Phone_Type': 'work', 'Phone': [nan]}, {'Phone_Type': 'home', 'Phone': [2177218280.0]}], []]

Gwang-Jin Kim · Accepted Answer · 2021-11-22 22:24:43Z

1

Write a function to dedupe lists:

def dedupe_list(lst):
    result = []
    for el in lst:
        if el not in result:
            result.append(el)
    return result

def dedupe_dict_values(dct):
    result = {}
    for key in dct:
        if type(dct[key]) is list:
            result[key] = dedupe_list(dct[key])
        else:
            result[key] = dct[key]
    return result

Test it:

deduped_dict = {'Customer_Number': 90617174, 
                'Email': [{'Email_Type': 'Primary', 
                           'Email': ['[email protected]']}], 
                'Phone_Number': [{'Phone_Type': 'Mobile', 
                                  'Phone': [12177218280]}]}


dedupe_dict_values(dct) == deduped_dict
## Out[12]: True

edited Nov 22, 2021 at 22:24

answered Nov 22, 2021 at 11:41

Gwang-Jin Kim

11k20 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Naveen Gupta Over a year ago

What is deduped_dict ?? getting error. I couldn't see if we are using anywhere

Gwang-Jin Kim Over a year ago

@NaveenGupta it was in your explanation. I inserted it again.

Collectives™ on Stack Overflow

Remove duplicates dictionaries from list of dictionaries in python

1 Answer 1

Test it:

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Test it:

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related