How to remove unhashable duplicates from a list in Python?

Question

My data is this:

[{u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': [u'/www/web'], u'server_port': u'80'}, {u'webpath': [u'/www/web'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/falvhezi'], u'server_port': u'80'}, {u'webpath': [u'/www/test10'], u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': [u'/www/400.ask.com'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/zhuanti'], u'server_port': u'80'}, {u'webpath': [u'/www/zhuanti'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}]

My code is this:

    seen = set()
    new_webpath_list = []
    for webpath in nginxConfs:
        t = tuple(webpath.items())
        if t not in seen:
            seen.add(t)
            new_webpath_list.append(webpath)

But the script returns:

TypeError: "unhashable type: 'list'"

possible duplicate of Remove duplicate dict in list in Python — muddyfish
– muddyfish, Commented Jul 23, 2015 at 13:48
@muddyfish it doesn't apply. the same error shall be raised. .items() doesn't turn list into tuple. — Jason Hu
– Jason Hu, Commented Jul 23, 2015 at 13:50
You are creating tuples, but there can still be un-hashable lists inside those tuples! — tobias_k
– tobias_k, Commented Jul 23, 2015 at 13:51

tobias_k · Accepted Answer · 2015-07-23 14:03:10Z

2

You are creating tuples from the dictionaries to make them hashable, but there can still be non-hashable lists inside those tuples! Instead, you also have to "tuplefy" the values.

t = tuple(((k, tuple(v)) for (k, v) in webpath.items()))

Note that this is a bit glitchy as the first entry in the dict is just a string, while the others are lists of strings. You could mend this with an if/else, but it should not really be necessary.

t = tuple(((k, tuple(v) if isinstance(v, list) else v) for (k, v) in webpath.items()))

Alternatively, you could also just memorize the string represenations of the dictionaries...

t = repr(webpath)

edited Jul 23, 2015 at 14:03

answered Jul 23, 2015 at 13:55

tobias_k

83.1k12 gold badges130 silver badges186 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

John Y · Accepted Answer · 2015-07-23 15:25:11Z

0

The most straightforward way to do this is to just test membership directly using the new list you are building.

new_webpath_list = []
for webpath in nginxConfs:
    if webpath not in new_webpath_list:
        new_webpath_list.append(webpath)

This handles the cases where there is an arbitrary (unknown beforehand) level of nesting of unhashable types. It also makes your code simpler, easier to understand, and very possibly more efficient, because you are not creating extra data that you don't need (no seen set, no conversion of elements to tuples).

edited Jul 23, 2015 at 15:25

answered Jul 23, 2015 at 14:03

John Y

14.6k2 gold badges51 silver badges76 bronze badges

Comments

Pedro Lobito · Accepted Answer · 2020-06-05 10:19:23Z

0

Late answer, but I was able to remove duplicated dict from a list using:

old_list = [{"x": 1}, {"x": 1}, {"x": 2}]
new_list = []
[new_list.append(x) for x in old_list if x not in new_list]
# [{'x': 1}, {'x': 2}]

Demo

edited Jun 5, 2020 at 10:19

answered Jun 5, 2020 at 1:44

Pedro Lobito

99.8k36 gold badges274 silver badges278 bronze badges

Collectives™ on Stack Overflow

How to remove unhashable duplicates from a list in Python?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related