How to Drop rows from pandas dataframe by using a list of indices

Question

Introduction

We have the following dataframe which we create from a CSV file.

data = pd.read_csv(path + name, usecols = ['QTS','DSTP','RSTP','DDATE','RDATE','DTIME','RTIME','DCXR','RCXR','FARE'])

I want to delete specific rows from the dataframe. For this purpose I used a list and appended the ids of the rows we want to delete.

for index,row in data.iterrows():
     if (row['FARE'] >= 2500.00):
       indices.append(index)

From here i am lost. Don't know how to use the ids in the list to delete the rows from the dataframe

Question

The list containing the row ids must be used in the dataframe to delete rows. Is it possible to do it?

Constraints

We can't use data.drop(index,inplace=True) because it really slows the process
We cannot use a filter because I have some special constraints.

Divakar · Accepted Answer · 2017-05-29 14:28:50Z

4

If you are trying to remove rows that have 'FARE' values greater than or equal to zero, you can use a mask that have those values lesser than 2500 -

df_out = df.loc[df.FARE.values < 2500] # Or df[df.FARE.values < 2500]

For large datasets, we might want to work with underlying array data and then construct the output dataframe -

df_out = pd.DataFrame(df.values[df.FARE.values < 2500], columns=df.columns)

To use those indices generated from the loopy code in the question -

df_out = df.loc[np.setdiff1d(df.index, indices)]

Or with masking again -

df_out = df.loc[~df.index.isin(indices)]  # or df[~df.index.isin(indices)]

edited May 29, 2017 at 14:28

answered May 29, 2017 at 13:55

Divakar

222k19 gold badges273 silver badges374 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Hani Gotc Over a year ago

thank you @Divakar but is there a way to use the ids as i stated in the question?

Divakar Over a year ago

@HaniGotc By ids, do you mean indices from the loop?

Hani Gotc Over a year ago

by using the list indices which contains the indices of the rows to delete

MaxU - stand with Ukraine · Accepted Answer · 2017-05-29 15:08:44Z

0

How about filtering data using DataFrame.query() method:

cols = ['QTS','DSTP','RSTP','DDATE','RDATE','DTIME','RTIME','DCXR','RCXR','FARE']
df = pd.read_csv(path + name, usecols=cols).query("FARE < 2500")

answered May 29, 2017 at 15:08

MaxU - stand with Ukraine

212k37 gold badges402 silver badges436 bronze badges

Collectives™ on Stack Overflow

How to Drop rows from pandas dataframe by using a list of indices

Introduction

Question

Constraints

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Introduction

Question

Constraints

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related