I'm trying to remove duplicate elements in column 'p1' and 'p2' i.e should the elements already occurred in column 'p1' it should not reappear in 'p2' or any subsequent column. For eg, for the code below, only 'a b' and 'c d' will remain.
Whats the effecient way of doing this?
import pandas as pd
df = pd.DataFrame({'p1':['a','b','a','a','b','d','c'],
'p2':['b','a','c','d','c','a','d'],
'value':[1,1,2,3,5,3,5]})
df
You can first set_index from column value, stack for creating Series, drop_duplicates, unstack and last reset_index:
print df.set_index('value').stack().drop_duplicates().unstack().reset_index()
value p1 p2
0 1 a b
1 2 None c
2 3 None d
print df.drop_duplicates(subset=['p1','p2']) ?df with None values?print pd.DataFrame({'p2': {1: 'a', 4: 'c'}, 'p1': {1: 'b', 4: 'b'}, 'value': {1: 1, 4: 5}}) ?Series( pd.DataFrame({'p1':['a','b','a','a','b','d','c'],'p2':['b','a','c','d','c','a','d'],'value':[1,1,2,3,5,3,5]}).values.ravel()).unique()
I'll post output as soon as I get pandas installed in my virtualenv.
p1 and p2. Next problem is lost index, if df is created by constructor.