Python Pandas: get rows of a DataFrame where a column is not null

Question

I'm filtering my DataFrame dropping those rows in which the cell value of a specific column is None.

df = df[df['my_col'].isnull() == False]

Works fine, but PyCharm tells me:

PEP8: comparison to False should be 'if cond is False:' or 'if not cond:'

But I wonder how I should apply this to my use-case? Using 'not ...' or ' is False' did not work. My current solution is:

df = df[df['my_col'].notnull()]

@MohammadAthar that doesn't work. As mentioned in my question. — Matthias
– Matthias, Commented Apr 5, 2018 at 13:19

FHTMitchell · Accepted Answer · 2018-04-05 13:34:29Z

37

So python has the short-circuiting logic operators not, and, or. These have a very specific meaning in python and cannot be overridden (not must return a bool and a and/or b always returns either a or b or throws an error.

However, python also has over-loadable boolean operators ~ (not), & (and), | (or) and ^ (xor).

You may recognise these as the int bitwise operators, but Numpy (and therefore pandas) use these to do array / series boolean operations.

For example

b = np.array([True, False, True]) & np.array([True, False, False])
# b --> [True False False]
b = ~b 
# b --> [False True True]

Hence what you want is

df = df[~df['my_col'].isnull()]

I agree with PEP8, don't do == False.

edited Apr 5, 2018 at 13:34

answered Apr 5, 2018 at 13:28

FHTMitchell

12.2k2 gold badges40 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Matthias Over a year ago

Thanks for the explantation and yet another example.

Collectives™ on Stack Overflow

Python Pandas: get rows of a DataFrame where a column is not null

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related