Pandas - select dataframe columns if statistic is greater than certain value

Question

I have pandas dataframe df. I would like to select columns which have standard deviation grater than 1. Here is what I tried

df2 = df[df.std() >1]
df2 = df.loc[df.std() >1]

Both generated error. What am I doing wrong?

You're trying to select from the row index, not the columns. Use instead: df.loc[:, df.std() > 1] — Alex Riley
– Alex Riley, Commented Aug 17, 2015 at 17:20

Vedansh Kedia · Accepted Answer · 2024-09-02 09:05:08Z

1

We need to get the list of columns whose values have standard deviation greater than 1.

That list of columns can then be passed to the dataframe to select the relevant data.

Be mindful to remove the columns of type "object" before trying to get the list. Below line get the list of columns.

df.columns[(df.std() > 1).to_list()]

Below line to get the dataframe with the selected columns.

df[df.columns[(df.std() > 1).to_list()]]

answered Sep 2, 2024 at 9:05

Vedansh Kedia

263 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

score 1 · Accepted Answer · 2018-02-20 17:24:31Z

1

Use df.loc[:, df.std() > 1] and it will fix it.

The first part which is [: refers to the rows and the second part df.std() > 1 refers to the columns

answered Feb 20, 2018 at 16:25

user9238790