Numpy NaN format in array not considered null

Question

I am trying to populate a dataframe with the following code:

df = pd.DataFrame(data=np.random.choice([1, np.nan], size=5))


0     1  
1     1  
2   NaN  
3     1  
4     1

Then:

df[df[0].isnull()]

2   NaN

So far, so good. But if I am modifying the 1 to '1' things get strange (imo).

df = pd.DataFrame(data=np.random.choice(['1', np.nan], size=5))

0    1  
1    1  
2    1  
3    1  
4  nan

Problems come with the isnull

df[df[0].isnull()]

Empty DataFrame  
Columns: [0]  
Index: []

How can I get the nan (which is a string) to behave like a NaN? I want to be able to filter quickly on all null/non-null values within my dataframe.

Thanks.

The issue here is that the NaN is being converted to the str nan which is surprising AFAIK real NaN require float dtype, in this case you'd have to compare with the str nan which is weird IMO — EdChum
– EdChum, Commented Dec 1, 2015 at 13:55
Thanks. For the moment I am converting the str nan back to the "normal" NaN . — Extratoro
– Extratoro, Commented Dec 1, 2015 at 14:59
I kinda feel this is a bug though as I wouldn't expect 'nan' when I've explicitly passed NaN — EdChum
– EdChum, Commented Dec 1, 2015 at 15:01

Peque · Accepted Answer · 2015-12-01 15:13:42Z

1

NaN is a concept which makes sense while working with numbers, not strings. When you create the dataframe with '1's Pandas is inferring the type of that column: str, which IMO is correct. So it will then convert NaN values to their string representation.

Note that if, for example, you say:

df = pd.DataFrame(data=np.random.choice(['1', 2], size=5))

The 2 will be converted as well to strings. Because, again, Pandas is inferring the string type for the whole column.

However, you can still filter easily with your proposed dataframe:

df = pd.DataFrame(data=np.random.choice(['1', np.nan], size=5))
df[df[0] == 'nan']

answered Dec 1, 2015 at 15:13

Peque

15.1k11 gold badges75 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Extratoro Over a year ago

I spot the automatic conversion, but it's not happening when using None. pd.DataFrame(data=np.random.choice(['1', None], size=5)) will generate a dataframe with some NoneType value in it.

Extratoro Over a year ago

Actually, playing with this, None is working with the isnull method.

Peque Over a year ago

@Extratoro: Interesting. Note that None is a different type; a basic Python type, which Pandas may be treating differently. However, NaN is a float type; see type(numpy.nan). So I guess Pandas will simply convert any number (int or float) to string when it infers the str type for the column (and NaN is simply a float type).

Collectives™ on Stack Overflow

Numpy NaN format in array not considered null

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related