Using str.contains on pandas dataframe [duplicate]

Question

This pandas python code generates the error message,

"TypeError: bad operand type for unary ~: 'float'"

I have no idea why because I'm trying to manipulate a str object

df_Anomalous_Vendor_Reasons[~df_Anomalous_Vendor_Reasons['V'].str.contains("File*|registry*")] #sorts, leaving only cases where reason is NOT File or Registry

Anybody got any ideas?

Can you post what happens with df_Anomalous_Vendor_Reasons['V'].str.contains("File*|registry*"), also do you need the asterisks here? — EdChum
– EdChum, Commented Jul 31, 2015 at 11:55
I can't seem to reproduce. Can you post df_Anomalous_Vendor_Reasons.to_msgpack() for us? — Mike Graham
– Mike Graham, Commented Jul 31, 2015 at 12:08
@MikeGraham: are you sure about that? regex=True is the default for str.contains in 0.16.2, anyway, and it seems to have used a regex compile for years. — DSM
– DSM, Commented Jul 31, 2015 at 12:56
Could it be something as simple as I have Nan values in my df? — Davtho1983
– Davtho1983, Commented Jul 31, 2015 at 13:22
Ok it was that. Sorry, I'm very new - I've been doing this a month and I've launched into pandas with no background even in python - so I need a lot of help — Davtho1983
– Davtho1983, Commented Jul 31, 2015 at 13:42

Josh · Accepted Answer · 2018-08-08 18:26:23Z

36

Credit to Davtho1983 comment above, I thought I'd add color to the comment for clarity.

For anyone stumbling on this later with the same error (like me). It's a very simple fix. The documentation from pandas shows

Series.str.contains(pat, case=True, flags=0, na=nan, regex=True)

What's happening is the contains() method isn't being applied to na values in the DataFrame, they will remain na. You just need to fill na values with Boolean values so you may use the invert operator ~ .

With the example above one should use

df_Anomalous_Vendor_Reasons[~df_Anomalous_Vendor_Reasons['V'].str.contains("File*|registry*", na=False)]

Of course one should choose False or True for the na argument based on intended logic. Whichever Boolean value you choose for filling na will be inverted.

edited Aug 8, 2018 at 18:26

answered Aug 5, 2016 at 3:11

Josh

2,8651 gold badge30 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

vagabond Over a year ago

goooood answer - i was quite confused by this

Chuck Over a year ago

Tremendous Answer

Collectives™ on Stack Overflow

Using str.contains on pandas dataframe [duplicate]

1 Answer 1

2 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Linked

Related