Select rows from a DataFrame based on presence of null value in specific column or columns

Question

I have an imported xls file as pandas dataframe, there are two columns containing coordinates which i will use to merge the dataframe with others which have geolocation data. df.info() shows 8859 records, the coordinatess columns have '8835 non-null float64' records.

I want to eyeball the 24 rows (that i assume are null) with all columns records to see if one of the other columns (street address town) can't be used to manually add back the coordinates for those 24 records. Ie. return dataframe for column in df.['Easting'] where isnull or NaN

I have adapted the method given here as below;

df.loc[df['Easting'] == NaN]

But get back an empty dataframe (0 rows × 24 columns), which makes no sense (to me). Attempting to use Null or Non null doesn't work as these values aren't defined. What am i missing?

Community · Accepted Answer · 2020-06-20 09:12:55Z

8

I think you need isnull for checking NaN values with boolean indexing:

df[df['Easting'].isnull()]

Docs:

Warning

One has to be mindful that in python (and numpy), the nan's don’t compare equal, but None's do. Note that Pandas/numpy uses the fact that np.nan != np.nan, and treats None like np.nan.

In [11]: None == None
Out[11]: True

In [12]: np.nan == np.nan
Out[12]: False

So as compared to above, a scalar equality comparison versus a None/np.nan doesn’t provide useful information.

In [13]: df2['one'] == np.nan
Out[13]: 
a    False
b    False
c    False
d    False
e    False
f    False
g    False
h    False
Name: one, dtype: bool

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Apr 24, 2016 at 7:45

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mapping dom Over a year ago

Thank you for the quick response, that worked a charm. I'm new to python and wasn't aware of the nan and none difference.

jezrael Over a year ago

Glad can help you! Good luck!

mapping dom Over a year ago

As an interesting aside to this i used this method to drop rows as part of the cleaning process as; NLUDData_Clean=NLUDData.dropna(subset = ['Easting']) Even though I have fewer records the size as returned by .info() query shows memory usage to have increased 1.6+mb to 1.7+mb not an issue for me but interested to know why for future

jezrael Over a year ago

Hard question for, maybe it is bug?

Collectives™ on Stack Overflow

Select rows from a DataFrame based on presence of null value in specific column or columns

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related