python: remove all rows in pandas dataframe that contain a string

Question

I've got a pandas dataframe called data and I want to remove all rows that contain a string in any column. For example, below we see the 'gdp' column has a string at index 3, and 'cap' at index 1.

data =

    y  gdp  cap
0   1    2    5
1   2    3    ab
2   8    7    2
3   3    bc   7
4   6    7    7
5   4    8    3
...

I've been trying to use something like this script because I will not know what is contained in exp_list ahead of time. Unfortunately, "data.var_name" throws out this error: 'DataFrame' object has no attribute 'var_name'. I also don't know what the strings will be ahead of time so is there anyway to generalize that as well?

exp_list = ['gdp', 'cap']

for var_name in exp_list:
    data = data[data.var_name != 'ab']

Acorbe · Accepted Answer · 2013-11-08 15:14:44Z

13

You can apply a function that tests row-wise your DataFrame for the presence of strings, e.g., say that df is your DataFrame

 rows_with_strings  = df.apply(
       lambda row : 
          any([ isinstance(e, basestring) for e in row ])
       , axis=1)

This will produce a mask for your DataFrame indicating which rows contain at least one string. You can hence select the rows without strings through the opposite mask

 df_with_no_strings = df[~rows_with_strings]

.

Example:

 a = [[1,2],['a',2], [3,4], [7,'d']]
 df = pd.DataFrame(a,columns = ['a','b'])


 df 
   a  b
0  1  2
1  a  2
2  3  4
3  7  d

select  = df.apply(lambda r : any([isinstance(e, basestring) for e in r  ]),axis=1) 

df[~select]                                                                                                                                

    a  b
 0  1  2
 2  3  4

edited Nov 8, 2013 at 15:14

answered Nov 8, 2013 at 14:34

Acorbe

8,4115 gold badges43 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Romain Jouin Over a year ago

is this row-wise of element wise ?

Acorbe Over a year ago

@romainjouin, It eliminates a row in case a string is present in any of its elements

JoeCondron · Accepted Answer · 2015-05-25 20:06:21Z

1

You can take the transpose, call ```convert_objects``, which works columns-wise, and then compare the data types to get a boolean key like this:

df[df.T.convert_objects().dtypes != object]

answered May 25, 2015 at 20:06

JoeCondron

8,9163 gold badges29 silver badges28 bronze badges

Collectives™ on Stack Overflow

python: remove all rows in pandas dataframe that contain a string

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related