Dealing with missing value in a column using pandas

Question

I am using the Auto MPG dataset which contains missing values in the column/attribute horsepower in the form of ? characters.

Hence when I use the code-

data.isnull.values.any()

OR

data["horsepower"].isnull.values.any()

Both of them return False since these codes work for NaN values or blank values.

How can I locate such missing values containing special character, which in my case happens to be ? rather than the traditional NaN value(s).

Thanks!

replace ? with NaN using df.horsepower.replace('?',np.NaN, inplace=True) Then go as usual — meW
– meW, Commented Jan 2, 2019 at 11:07
If you are reading your data from csv file then provide na_values as ? in read_csv. for more details visit pandas.pydata.org/pandas-docs/stable/generated/… — Mohamed Thasin ah
– Mohamed Thasin ah, Commented Jan 2, 2019 at 11:08
@MohamedThasinah I've verified my answer before posting. It's working. — meW
– meW, Commented Jan 2, 2019 at 11:15

jezrael · Accepted Answer · 2019-01-02 11:08:27Z

3

Use replace before checking NaNs:

data["horsepower"].replace('?',np.nan).isnull().values.any()

If DataFrame is created by read_csv add parameter na_values for converting ? to NaNs:

data = pd.read_csv(path, na_values=["?"])
data["horsepower"].isnull().values.any()

answered Jan 2, 2019 at 11:08

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

patelnisheet · Accepted Answer · 2019-01-02 12:29:07Z

2

you need to convert ? to NaN first. After then You can go for finding null values in it.

1) to convert ? to NaN :

data.replace('?',np.NaN)

2) to find null values:

pd.isna(data['horsepower'])

it will return dataframe with series of True/False.

answered Jan 2, 2019 at 12:29

patelnisheet

1441 gold badge3 silver badges12 bronze badges

Comments

anky · Accepted Answer · 2019-01-02 11:18:10Z

2

you can define na_values as ? or use the below:

df.replace(r'[\W]',np.nan,regex=True)

\W finds any character that is not a letter, numeric digit, or the underscore character.

answered Jan 2, 2019 at 11:18

anky

75.3k11 gold badges46 silver badges76 bronze badges

Collectives™ on Stack Overflow

Dealing with missing value in a column using pandas

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related