Pandas dropping duplicates doesn't drop last duplicate

Question

Setting keep=False should remove all duplicates but if I run my function is still returns a duplicate of the previous row

def date_to_csv():
   import pandas as pd
   from random import randint
   df = pd.read_csv("test.csv")
   df = df.append({'Date': datetime.date.today(), 'Price': randint(1,100)}, ignore_index=True)
   result_df = df.drop_duplicates(keep=False)
   result_df.to_csv('test.csv', mode='a', index=False, header=None)

If my csv file is empty with only the column headers 'Date' and 'Price' and I run my function 3 times it returns this in csv:

Date,Price
2021-06-26,74
2021-06-26,74
2021-06-26,51
2021-06-26,51
2021-06-26,13

When I expect it to return something like this:

Date,Price
2021-06-26,74
2021-06-26,51
2021-06-26,13

Are there other fields in your test.csv?

forgetso
– forgetso

2021-06-26 10:40:48 +00:00
Commented Jun 26, 2021 at 10:40 — forgetso
– forgetso, Commented Jun 26, 2021 at 10:40
only the two column headers 'Date' and 'Price'

buckenup
– buckenup

2021-06-26 10:45:48 +00:00
Commented Jun 26, 2021 at 10:45 — buckenup
– buckenup, Commented Jun 26, 2021 at 10:45

Vladyslav Dusiak · Accepted Answer · 2021-06-26 11:06:21Z

2

Because of mode='a' you can't remove previous duplicates after several execution of your function. Here is a code for your expected behaviour:

import pandas as pd
from datetime import datetime


def date_to_csv(): 
     df = pd.read_csv('test.csv') 
     df = df.append({'Date': str(datetime.now().date()), 'Price': randint(1, 100)}, ignore_index=True) 
     df.to_csv('test.csv', index=False)

edited Jun 26, 2021 at 11:06

answered Jun 26, 2021 at 10:57

Vladyslav Dusiak

615 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Cimbali Over a year ago

Also the last Date value is a datetime object, the previous ones are strings.

Collectives™ on Stack Overflow

Pandas dropping duplicates doesn't drop last duplicate

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related