Python Pandas out of bounds datetime timestamp error for long dataframe

Question

I have a data frame df with shape (500000,70) and several columns including invalid dates like 4000-01-01 00:00:00. In a smaller version of this data frame I tried

df["date"] = df["date"].astype(str)
df["date"] = df["date"].replace('4000-01-01 00:00:00', pd.NaT)

which worked fine. Also the version

df["date"] = pd.to_datetime(df["date"].replace("4000-01-01 00:00:00",pd.NaT))

worked. For the long data frame version I receive the following error

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 4000-01-01 00:00:00

Any suggestions how to solve this problem in an elegant way or what the problem might be?

Thank you.

jezrael · Accepted Answer · 2018-10-17 06:10:37Z

2

If add parameter errors='coerce' to to_datetime function it return NaT for all not parseable datetimes:

df["date"] = pd.to_datetime(df["date"], errors='coerce')

answered Oct 17, 2018 at 6:10

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mayank Porwal · Accepted Answer · 2018-10-17 06:24:39Z

1

The error is because:

In [332]: pd.Timestamp.max
Out[332]: Timestamp('2262-04-11 23:47:16.854775807')

The upper limit of the date is this. And your value is out of the range, hence OutOfBoundsError.

edited Oct 17, 2018 at 6:24

answered Oct 17, 2018 at 6:11

Mayank Porwal

34.2k9 gold badges45 silver badges65 bronze badges

Collectives™ on Stack Overflow

Python Pandas out of bounds datetime timestamp error for long dataframe

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related