218

I have a list in Python, how can I make it's values unique?

Improve this question
5
  • 1
    Or this if you want to preserve the ordering. Commented Dec 16, 2010 at 10:31
  • Please fix the title of your question. You're not talking about make lists distinct. You're talking about making list items distinct. Commented Dec 16, 2010 at 11:31
  • 1
    Why do you need list in the first place? Maybe set() or dict() are enough. Commented Dec 16, 2010 at 14:22
  • Also see here for more information Commented Mar 14, 2014 at 17:37
  • 5
    Possible duplicate of Removing duplicates in lists or stackoverflow.com/questions/480214/… Commented Dec 18, 2017 at 14:25

10 Answers 10

Reset to default
404

The simplest is to convert to a set then back to a list:

my_list = list(set(my_list))

One disadvantage with this is that it won't preserve the order. You may also want to consider if a set would be a better data structure to use in the first place, instead of a list.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

i am wrong or with python3k the values will be preserved, cause set now are sorted?
@Ant Dictionary key order is preserved from Python 3.6, but it says "the order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon". Since they're both based on hashes, I'd think set would be the same, but it's not mentioned, so apparently not: docs.python.org/3.6/whatsnew/3.6.html
Preserve order and functional way: In [23]: from functools import reduce In [24]: reduce(lambda acc,elem: acc+[elem] if not elem in acc else acc , [2,1,2,3,3,3,4,5], []) Out[24]: [2, 1, 3, 4, 5]
The order of the list gets lost this way
worth mentioning that this doesn't work if the list contains a list.
34

Modified versions of http://www.peterbe.com/plog/uniqifiers-benchmark

To preserve the order:

def f(seq): # Order preserving
  ''' Modified version of Dave Kirby solution '''
  seen = set()
  return [x for x in seq if x not in seen and not seen.add(x)]

OK, now how does it work, because it's a little bit tricky here if x not in seen and not seen.add(x):

In [1]: 0 not in [1,2,3] and not print('add')
add
Out[1]: True

Why does it return True? print (and set.add) returns nothing:

In [3]: type(seen.add(10))
Out[3]: <type 'NoneType'>

and not None == True, but:

In [2]: 1 not in [1,2,3] and not print('add')
Out[2]: False

Why does it print 'add' in [1] but not in [2]? See False and print('add'), and doesn't check the second argument, because it already knows the answer, and returns true only if both arguments are True.

More generic version, more readable, generator based, adds the ability to transform values with a function:

def f(seq, idfun=None): # Order preserving
  return list(_f(seq, idfun))

def _f(seq, idfun=None):  
  ''' Originally proposed by Andrew Dalke '''
  seen = set()
  if idfun is None:
    for x in seq:
      if x not in seen:
        seen.add(x)
        yield x
  else:
    for x in seq:
      x = idfun(x)
      if x not in seen:
        seen.add(x)
        yield x

Without order (it's faster):

def f(seq): # Not order preserving
  return list(set(seq))
Improve this answer

1 Comment

sort of inner helper function (there was a bug in the code, should be _f instead of _f10 on line 2, thanks for spotting)
31

one-liner and preserve order

list(OrderedDict.fromkeys([2,1,1,3]))

although you'll need

from collections import OrderedDict
Improve this answer

3 Comments

An alternative form is: OrderedDict.fromkeys(my_list).keys()
@DannyStaple: that works in python 2, but in python 3 it returns a view of the dictionary keys, which might be okay for some purposes, but doesn't support indexing for example.
The initial one liner will work. The aternative form returns an odict_keys type, which is less useful for this - but can still be converted to a list.
18

Let me explain to you by an example:

if you have Python list

>>> randomList = ["a","f", "b", "c", "d", "a", "c", "e", "d", "f", "e"]

and you want to remove duplicates from it.

>>> uniqueList = []

>>> for letter in randomList:
    if letter not in uniqueList:
        uniqueList.append(letter)

>>> uniqueList
['a', 'f', 'b', 'c', 'd', 'e']

This is how you can remove duplicates from the list.

Improve this answer

2 Comments

+1 because it's the only one that works for types that are unhashable, but do have an eq function (if your types are hashable, use one of the other solutions). Note that it will be slow for very big lists.
Unless in some special case as Claude explained, this one has the worst performance: O(n^2)
15

To preserve the order:

l = [1, 1, 2, 2, 3]
result = list()
map(lambda x: not x in result and result.append(x), l)
result
# [1, 2, 3]
Improve this answer

3 Comments

In python 3.4 returns an empty list!!!
map just creates map object (generator), does not execute it. list(map(....)) forces the execution
It will produce not optimal performance, due to traversal the result list for each x.
11

How about dictionary comprehensions?

>>> mylist = [3, 2, 1, 3, 4, 4, 4, 5, 5, 3]

>>> {x:1 for x in mylist}.keys()
[1, 2, 3, 4, 5]

EDIT To @Danny's comment: my original suggestion does not keep the keys ordered. If you need the keys sorted, try:

>>> from collections import OrderedDict

>>> OrderedDict( (x,1) for x in mylist ).keys()
[3, 2, 1, 4, 5]

which keeps elements in the order by the first occurrence of the element (not extensively tested)

Improve this answer

2 Comments

This would not preserve order - dictionary order (and set order) is determined by the hashing algorithm and not insertion order. I am not sure of the effects of a dictionary comprehension with an OrderedDict type though.
@DannyStaple True. I added an example using OrderedDict and a generator, if ordered output is desired.
5

The characteristics of sets in Python are that the data items in a set are unordered and duplicates are not allowed. If you try to add a data item to a set that already contains the data item, Python simply ignores it.

>>> l = ['a', 'a', 'bb', 'b', 'c', 'c', '10', '10', '8','8', 10, 10, 6, 10, 11.2, 11.2, 11, 11]
>>> distinct_l = set(l)
>>> print(distinct_l)
set(['a', '10', 'c', 'b', 6, 'bb', 10, 11, 11.2, '8'])
Improve this answer

Comments

4

If all elements of the list may be used as dictionary keys (i.e. they are all hashable) this is often faster. Python Programming FAQ

d = {}
for x in mylist:
    d[x] = 1
mylist = list(d.keys())
Improve this answer

Comments

2

From http://www.peterbe.com/plog/uniqifiers-benchmark:

def f5(seq, idfun=None):  
    # order preserving
    if idfun is None:
        def idfun(x): return x
    seen = {}
    result = []
    for item in seq:
        marker = idfun(item)
        # in old Python versions:
        # if seen.has_key(marker)
        # but in new ones:
        if marker in seen: continue
        seen[marker] = 1
        result.append(item)
    return result
Improve this answer

3 Comments

Wouldn't it make sense to use a set for seen, rather than a dict?
In Python, sets and dicts are built using hashtables so they are interchangeable in this scenario. They both provide the same operations (limiting duplicates) and both have the same running time.
This one is slow, generator version is much faster
2

The simplest way to remove duplicates whilst preserving order is to use collections.OrderedDict (Python 2.7+).

from collections import OrderedDict
d = OrderedDict()
for x in mylist:
    d[x] = True
print d.iterkeys()
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.