I need to remove duplicate rows from a dataset. Basically, I should perform
proc sort data=mydata noduprecs dupout=mydata_dup;run;
I need to remove duplicates as well as save those duplicate rows in a separate dataframe. How can I do that?
Add a comment
|
2 Answers
Assuming your dataset is a pandas dataframe.
To remove the duplicated rows:
data = data.drop_duplicates()
To select all the duplicated rows:
dup = data.ix[data.duplicated(), :]
Hope it helps.
Comments
A few examples from Pandas docs:
> df = pd.DataFrame({
'brand': ['Yum Yum', 'Yum Yum', 'Indomie', 'Indomie', 'Indomie'],
'style': ['cup', 'cup', 'cup', 'pack', 'pack'],
'rating': [4, 4, 3.5, 15, 5]
})
> df
brand style rating
0 Yum Yum cup 4.0
1 Yum Yum cup 4.0
2 Indomie cup 3.5
3 Indomie pack 15.0
4 Indomie pack 5.0
By default, it removes duplicate rows based on all columns.
> df.drop_duplicates()
brand style rating
0 Yum Yum cup 4.0
2 Indomie cup 3.5
3 Indomie pack 15.0
4 Indomie pack 5.0
To remove duplicates on specific column(s), use subset.
> df.drop_duplicates(subset=['brand'])
brand style rating
0 Yum Yum cup 4.0
2 Indomie cup 3.5
To remove duplicates and keep last occurrences, use keep.
> df.drop_duplicates(subset=['brand', 'style'], keep='last')
brand style rating
1 Yum Yum cup 4.0
2 Indomie cup 3.5
4 Indomie pack 5.0
