Remove rows with empty lists from pandas data frame

Question

I have a data frame with some columns with empty lists and others with lists of strings:

       donation_orgs                              donation_context
0            []                                           []
1   [the research of Dr. ...]   [In lieu of flowers , memorial donations ...]

I'm trying to return a data set without any of the rows where there are empty lists.

I've tried just checking for null values:

dfnotnull = df[df.donation_orgs != []]
dfnotnull

and

dfnotnull = df[df.notnull().any(axis=1)]
pd.options.display.max_rows=500
dfnotnull

And I've tried looping through and checking for values that exist, but I think the lists aren't returning Null or None like I thought they would:

dfnotnull = pd.DataFrame(columns=('donation_orgs', 'donation_context'))
for i in range(0,len(df)):
    if df['donation_orgs'].iloc(i):
        dfnotnull.loc[i] = df.iloc[i]

All three of the above methods simply return every row in the original data frame.=

In my experience it is quite perilous to keep data in lists within data frames. It can make grouping and aggregation functions go wrong. If you must do it, consider the tuple instead, that seems to work better. — Woody Pride
– Woody Pride, Commented Dec 8, 2015 at 18:50

wjandrea · Accepted Answer · 2024-07-11 16:49:41Z

86

To avoid converting to str and actually use the lists, you can do this:

df[df['donation_orgs'].map(len) > 0]

It maps the donation_orgs column to the length of the lists of each row and keeps only the ones that have at least one element, filtering out empty lists.

It returns

Out[1]: 
                            donation_context          donation_orgs
1  [In lieu of flowers , memorial donations]  [the research of Dr.]

as expected.

edited Jul 11, 2024 at 16:49

wjandrea

33.8k10 gold badges69 silver badges105 bronze badges

answered Mar 7, 2018 at 9:58

Victor

3,6312 gold badges22 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Leothorn Over a year ago

this should be the accepted answer . Its more elegant

MrKsn Over a year ago

df[df['donation_orgs'].map(len) > 0], or even df[df['donation_orgs'].map(bool)]

Sameer Girolkar Over a year ago

df[df['donation_orgs'].map(bool)] this works best as this can even handle null values

Woody Pride · Accepted Answer · 2015-12-08 18:23:27Z

38

You could try slicing as though the data frame were strings instead of lists:

import pandas as pd
df = pd.DataFrame({
'donation_orgs' : [[], ['the research of Dr.']],
'donation_context': [[], ['In lieu of flowers , memorial donations']]})

df[df.astype(str)['donation_orgs'] != '[]']

Out[9]: 
                            donation_context          donation_orgs
1  [In lieu of flowers , memorial donations]  [the research of Dr.]

answered Dec 8, 2015 at 18:23

Woody Pride

14k10 gold badges51 silver badges64 bronze badges

Comments

Amir Imani · Accepted Answer · 2018-02-13 23:01:39Z

12

You can use the following one-liner:

df[(df['donation_orgs'].str.len() != 0) | (df['donation_context'].str.len() != 0)]

answered Feb 13, 2018 at 23:01

Amir Imani

3,2552 gold badges25 silver badges27 bronze badges

Comments

Mark · Accepted Answer · 2020-01-16 17:21:54Z

5

Assuming that you read data from a CSV, the other possible solution could be this:

import pandas as pd

df = pd.read_csv('data.csv', na_filter=True, na_values='[]')
df.dropna()

na_filter defines additional string to recognize as NaN. I tested this on pandas-0.24.2.

answered Jan 16, 2020 at 17:21

Mark

6,60412 gold badges79 silver badges157 bronze badges

Comments

cottontail · Accepted Answer · 2022-07-15 07:16:15Z

1

It's probably that the data type is different, This will help probably

df[df.astype(str)['donation_orgs'] != '[]']

edited Jul 15, 2022 at 7:16

cottontail

25.5k25 gold badges184 silver badges176 bronze badges

answered Jul 14, 2022 at 9:30

Mohsin Khan

214 bronze badges

Collectives™ on Stack Overflow

Remove rows with empty lists from pandas data frame

5 Answers 5

3 Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related