Pandas drop duplicates across columns

Question

Good day,

I need a way to check each row of a dataframe and drop the row if all the values in that row (across the score columns) are the same. The person_id may differ.

Here is a part of the dataset:

In:

data = [[7, 10, 10, 10, 10], [17, 10, 10, 10, 10], [18, 8, 10, 10, 10], [20, 10, 10, 9, 9], [25, 9, 8, 8, 7]]
df = pd.DataFrame(data, columns = ['person_id', 'score_1', 'score_2', 'score_3', 'score_4'])
df

Out:

    person_id   score_1 score_2 score_3 score_4
0   7           10      10      10      10
1   17          10      10      10      10
2   18          8       10      10      10
3   20          10      10      9       9
4   25          9       8       8       7

The desired output would be:

    person_id   score_1 score_2 score_3 score_4
2   18          8       10      10      10
3   20          10      10      9       9
4   25          9       8       8       7

Since row 0 (person_id 7) and row 1 (person_id 17) have the same scores. The number of columns will also change, adding more score columns - thus, I cannot use

df_no_duplicates = df.loc[(df.score_1 != df.score_2) | (df.score_2 != df.score_3)| (df.score_3 != df.score_4)]

timgeb · Accepted Answer · 2022-02-09 15:13:59Z

1

A cheeky way to omit the person_id column from the process is to set it as the index temporarily.

result = df.set_index('person_id').drop_duplicates(keep=False).reset_index()

You could also use the subset argument of drop_duplicates:

result = df.drop_duplicates(subset=df.columns.difference(['person_id']), keep=False)

answered Feb 9, 2022 at 15:13

timgeb

79.2k20 gold badges129 silver badges150 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

BENY · Accepted Answer · 2022-02-09 15:12:11Z

1

You can try nunique

out = df[df.filter(like='score').nunique(1)>1].copy()
Out[208]: 
   person_id  score_1  score_2  score_3  score_4
2         18        8       10       10       10
3         20       10       10        9        9
4         25        9        8        8        7

answered Feb 9, 2022 at 15:12

BENY

324k22 gold badges176 silver badges250 bronze badges

Collectives™ on Stack Overflow

Pandas drop duplicates across columns

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related