Pandas Assert error for identical columns

Question

Trying to assert two pandas dataframe using.

pd.testing.assert_frame_equal(df, expected_df, check_dtype=False)

Getting the error as below. But if you see properly, both left and right have the exact same value.

AssertionError: DataFrame.iloc[:, 9] (column name="Stores") are different

DataFrame.iloc[:, 9] (column name="Stores") values are different (100.0 %)
[left]:  [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, ...]
[right]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, ...]

Firstly, if you're checking identical dataframes, why is the check_dype parameter set to False? Secondly, could you show us how you're generating the data to populate these dataframes? — Dbercules
– Dbercules, Commented Feb 8, 2021 at 14:52
I am doing the unittest to verify if my deployments have made any changes in the results. df is the results that is stored in postgres by my code. df - I am pulling from postgres DB. Its a simple select * query expected_df - I had pulled from postgres before deployment of some codes. I am trying to do a unittest to confirm if my codes have made any affect to the results now. — Rakesh SK
– Rakesh SK, Commented Feb 8, 2021 at 14:53
Does pandas.DataFrame.equals give you the same result? Why is check_dtype set to False , what are the dtypes? How about the indexes, are they the same? What does df['Stores'] - df_expected['Stores'] return? Have you played with the other parameters as explained in the docs? pandas.pydata.org/pandas-docs/stable/reference/api/… — Pythonista anonymous
– Pythonista anonymous, Commented Feb 8, 2021 at 14:54
@RakeshSK, if df is populated via that query, where is expected_df populated from? As we can't see your data, we are limited in our ability to help more than provide these suggestions. — Dbercules
– Dbercules, Commented Feb 8, 2021 at 14:57
@Dbercules Both are from the derived from the same DB. Its just that expected_df was downloaded and kept as CSV in local before deployment. df was downloaded from postgres after code deployment — Rakesh SK
– Rakesh SK, Commented Feb 8, 2021 at 15:07

Marcin Baranek · Accepted Answer · 2022-10-12 12:29:24Z

1

Check that the column types are the same. I had a similar problem when I was reading data from a postgresql table and the int column was declared as text. Then after reading the data from the database and writing it to a csv file, a column of type string with int values is written the same as a column of type int. After reading the same file, this column automatically gets type int. As string and int are not comparable it returns AsserationError

answered Oct 12, 2022 at 12:29

Marcin Baranek

664 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nikita Shabankin Over a year ago

This does not really answer the question. If you have a different question, you can ask it by clicking Ask Question. To get notified when this question gets new answers, you can follow this question. Once you have enough reputation, you can also add a bounty to draw more attention to this question. - From Review

Collectives™ on Stack Overflow

Pandas Assert error for identical columns

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related