How to drop columns and rows with missing values?

Question

I've been trying to take a pandas.Dataframe and drop its rows and columns with missing values simultaneously. While trying to use dropna and applying on both axis, I found out that this is no longer supported. So then I tried, using dropna, to drop the columns and then drop the rows and vice versa and obviously, the results come out different as the values no longer reflect the initial state accurately. So to give an example I receive:

pandas.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
                   "toy": [numpy.nan, 'Batmobile', 'Bullwhip'],
                   "weapon": [numpy.nan, 'Boomerang', 'Gun']})

and return:

pandas.DataFrame({"name": ['Batman', 'Catwoman']})

Any help will be appreciated.

jezrael · Accepted Answer · 2022-04-29 07:59:34Z

1

Test if all values per columns and for rows use DataFrame.notna with DataFrame.any and DataFrame.loc:

m = df.notna()
df0 = df.loc[m.all(1), m.all()]
print (df0)
      name
1    Batman
2  Catwoman

edited Apr 29, 2022 at 7:59

answered Apr 29, 2022 at 7:51

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SiiilverSurfer Over a year ago

How would you apply this solution if it's needed to drop columns and rows only where all their values are missing?

jezrael Over a year ago

@EliranZeitouni - simpliest is df = df.dropna(how='all', axis=0) or df = df.dropna(how='all', axis=1)

Collectives™ on Stack Overflow

How to drop columns and rows with missing values?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related