Python Remove duplicates and original from nested list based on specific key

Question

I m trying to delete all duplicates & original from a nested list based on specific column.

Example

list = [['abc',3232,'demo text'],['def',9834,'another text'],['abc',0988,'another another text'],['poi',1234,'text']]

The key column is the first (abc, def, abc) and based on this I want to remove any item (plus the original) which has the same value with the original.

So the new list should contain:

newlist = [['def',9834,'another text'],['poi',1234,'text']]

I found many similar topics but not for nested lists... Any help please?

Side point. Never name a variable after a built-in, use L or list_ instead of list. — jpp
– jpp, Commented Jun 15, 2018 at 9:28

taras · Accepted Answer · 2018-06-15 08:30:14Z

2

You can construct a list of keys

keys = [x[0] for x in list]

and select only those records for which the key occurs exactly once

newlist = [x for x in list if keys.count(x[0]) == 1]

answered Jun 15, 2018 at 8:30

taras

6,93510 gold badges46 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jpp Over a year ago

You have O(n^2) complexity here by calling list.count n times. You could use collections.Counter to make this O(n). Or store your counts separately.

taras Over a year ago

Well, OP didn't say anything regarding the list size, so I assumed it is not large enough to make the difference between O(n) and O(n^2). Definitely, using Counter is more efficient approach but I intended to give a quick-and-dirty solution that works well in most cases.

jpp Over a year ago

My comment isn't a complaint, it's just a note which may interest readers.

taras Over a year ago

No offense taken;) Just explained my intent.

Austin · Accepted Answer · 2018-06-15 09:37:11Z

1

Use collections.Counter:

from collections import Counter

lst = [['abc',3232,'demo text'],['def',9834,'another text'],['abc',988,'another another text'],['poi',1234,'text']]

d = dict(Counter(x[0] for x in lst))
print([x for x in lst if d[x[0]] == 1])

# [['def', 9834, 'another text'], 
#  ['poi', 1234, 'text']]

Also note that you shouldn't name your list as list as it shadows the built-in list.

edited Jun 15, 2018 at 9:37

answered Jun 15, 2018 at 8:41

Austin

26.1k4 gold badges28 silver badges52 bronze badges

2 Comments

jpp Over a year ago

Good solution, this has O(n) complexity. But I don't think if x[0] in d.keys() is necessary?

Austin Over a year ago

@jpp oops! That isn't necessary. Thanks a lot.

Rakesh · Accepted Answer · 2018-06-15 08:28:34Z

1

Using a list comprehension.

Demo:

l = [['abc',3232,'demo text'],['def',9834,'another text'],['abc', 988,'another another text'],['poi',1234,'text']]
checkVal = [i[0] for i in l]
print( [i for i in l if not checkVal.count(i[0]) > 1 ] )

Output:

[['def', 9834, 'another text'], ['poi', 1234, 'text']]

answered Jun 15, 2018 at 8:28

Rakesh

82.9k17 gold badges85 silver badges122 bronze badges

1 Comment

jpp Over a year ago

You have O(n^2) complexity here by calling list.count n times. You could use collections.Counter to make this O(n). Or store your counts separately.

jpp · Accepted Answer · 2018-06-15 08:31:11Z

1

Using collections.defaultdict for an O(n) solution:

L = [['abc',3232,'demo text'],
     ['def',9834,'another text'],
     ['abc',988,'another another text'],
     ['poi',1234,'text']]

from collections import defaultdict

d = defaultdict(list)

for key, num, txt in L:
    d[key].append([num, txt])

res = [[k, *v[0]] for k, v in d.items() if len(v) == 1]

print(res)

[['def', 9834, 'another text'],
 ['poi', 1234, 'text']]

answered Jun 15, 2018 at 8:31

jpp

166k37 gold badges301 silver badges362 bronze badges

1 Comment

Austin Over a year ago

Between this solution is also good. I usually go for Counter than a defaultdict way. +1.

Collectives™ on Stack Overflow

Python Remove duplicates and original from nested list based on specific key

4 Answers 4

4 Comments

2 Comments

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

4 Comments

2 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related