Dealing with missing data in Pandas and Numpy

Question

I have the following data sample. I would like to

a) in column C, replace the np.NaN with 999,
b) in column D, place '' with np.NaN.

Neither of my attempts is working, and I am not sure why.

import pandas
from pandas import DataFrame
import numpy as np


df = DataFrame({'A' : ['foo', 'foo', 'foo', 'foo',
                        'bar', 'bar', 'bar', 'bar'],
                 'B' : ['one', 'one', 'two', 'three',
                        'two', 'two', 'one', 'three'],
                 'C' : [1, np.NaN, 1, 2, np.NaN, 1, 1, 2], 'D' : [2, '', 1, 1, '', 2, 2, 1]})

print df

df.C.fillna(999)
df.D.replace('', np.NaN)

print df

Output: 

 A      B   C  D
0  foo    one   1  2
1  foo    one NaN   
2  foo    two   1  1
3  foo  three   2  1
4  bar    two NaN   
5  bar    two   1  2
6  bar    one   1  2
7  bar  three   2  1
     A      B   C  D
0  foo    one   1  2
1  foo    one NaN   
2  foo    two   1  1
3  foo  three   2  1
4  bar    two NaN   
5  bar    two   1  2
6  bar    one   1  2
7  bar  three   2  1

EdChum · Accepted Answer · 2015-09-14 13:47:50Z

Those operations return a copy of the data (most of the pandas ops behave the same), they don't operate in place unless you explicitly say so (the default is inplace=False), see fillna and replace:

df.C.fillna(999, inplace=True)
df.D.replace('', np.NaN, inplace=True)

or assign back:

df['C'] = df.C.fillna(999)
df['D'] = df.D.replace('', np.NaN)

Also I strongly suggest you access your columns using subscript operator [] rather than as an attribute using dot operator . to avoid ambiguous behaviour

In [60]:
df = pd.DataFrame({'A' : ['foo', 'foo', 'foo', 'foo',
                        'bar', 'bar', 'bar', 'bar'],
                 'B' : ['one', 'one', 'two', 'three',
                        'two', 'two', 'one', 'three'],
                 'C' : [1, np.NaN, 1, 2, np.NaN, 1, 1, 2], 'D' : [2, '', 1, 1, '', 2, 2, 1]})

df.C.fillna(999, inplace =True)
df.D.replace('', np.NaN, inplace=True)
df

Out[60]:
     A      B    C   D
0  foo    one    1   2
1  foo    one  999 NaN
2  foo    two    1   1
3  foo  three    2   1
4  bar    two  999 NaN
5  bar    two    1   2
6  bar    one    1   2
7  bar  three    2   1

Collectives™ on Stack Overflow

Dealing with missing data in Pandas and Numpy

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related