How to select certain rows of a dataframe in Python?

Question

I have a pandas dataframe that looks like this:

              x_cor
y_cor             
893.200012       1
893.299988      17
893.400024      41
893.500000      39
893.599976      40
893.700012      36
893.799988       2
893.900024      13
894.000000      44
894.099976      43
894.200012      74
894.299988      88
894.400024      78
894.500000     132
894.599976     180
894.700012     178

What I want to do is to select certain rows, according to a condition, and to create 2 different dataframes out of it(one of which is comprised of the rows that meet the condition and the other of which consists of the rows that don't meet the condition). The condition is whether or not the x_cor value of each row is bigger than the preceding and subseqent x_cor values.

For example, the 3rd row 893.400024 41 meets the condition because the previous row's x_cor is 17 and the next row's x_cor is 39, which are smaller than 41.

I think it would be inefficient if I used loops with iloc or ix. What would be a better way to do this?

BENY · Accepted Answer · 2018-05-18 15:31:22Z

3

From scipy argrelextrema

from scipy.signal import argrelextrema
df.iloc[argrelextrema(df.x_cor.values, np.greater)]
Out[981]: 
            x_cor
y_cor            
893.400024     41
893.599976     40
894.000000     44
894.299988     88
894.599976    180

answered May 18, 2018 at 15:31

BENY

324k22 gold badges176 silver badges250 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

maynull Over a year ago

@Wen Thank you for your help! Do you happen to know how to get a dataframe the rows of which don't meet the condition?

BENY Over a year ago

@maynull df.iloc[~df.index.isin(argrelextrema(df.x_cor.values, np.greater)[0])]

cs95 Over a year ago

Fantastic answer. I've already upvoted (upvoted both actually)

maynull Over a year ago

@Wen Thank you so much! Scipy is great!

cs95 Over a year ago

Me too :-( By the way, decided it was for the best; too many people had issues pinging me, I see \@COLDSPEED everywhere instead of \@cᴏʟᴅsᴘᴇᴇᴅ like it should be >:-(

user3483203 · Accepted Answer · 2018-05-18 15:27:03Z

3

Use shift

df.loc[(df.x_cor > df.x_cor.shift(1)) & (df.x_cor > df.x_cor.shift(-1))]

         y_cor  x_cor
2   893.400024     41
4   893.599976     40
8   894.000000     44
11  894.299988     88
14  894.599976    180

answered May 18, 2018 at 15:27

user3483203

51.3k10 gold badges72 silver badges104 bronze badges

2 Comments

maynull Over a year ago

Wow, it's such a simple and still powerful way to do the job! Thank you for your help!

user3483203 Over a year ago

@maynull use Wen's solution, much faster than mine

Collectives™ on Stack Overflow

How to select certain rows of a dataframe in Python?

2 Answers 2

5 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related