Check if a string in a Pandas DataFrame column is in a list of strings

Question

If I have a frame like this

frame = pd.DataFrame({
    "a": ["the cat is blue", "the sky is green", "the dog is black"]
})

and I want to check if any of those rows contain a certain word I just have to do this.

frame["b"] = (
   frame.a.str.contains("dog") |
   frame.a.str.contains("cat") |
   frame.a.str.contains("fish")
)

frame["b"] outputs:

0     True
1    False
2     True
Name: b, dtype: bool

If I decide to make a list:

mylist = ["dog", "cat", "fish"]

How would I check that the rows contain a certain word in the list?

The method in the accepted answer will find, for example, substring 'the' in a word 'there'. For a method finding exact words see here: Creating a new column by finding exact word in a column of strings — Georgy
– Georgy, Commented Apr 11, 2018 at 8:52

Janosh · Accepted Answer · 2021-05-11 00:20:31Z

140

frame = pd.DataFrame({'a' : ['the cat is blue', 'the sky is green', 'the dog is black']})

frame
                  a
0   the cat is blue
1  the sky is green
2  the dog is black

The str.contains method accepts a regular expression pattern:

mylist = ['dog', 'cat', 'fish']
pattern = '|'.join(mylist)

pattern
'dog|cat|fish'

frame.a.str.contains(pattern)
0     True
1    False
2     True
Name: a, dtype: bool

Because regex patterns are supported, you can also embed flags:

frame = pd.DataFrame({'a' : ['Cat Mr. Nibbles is blue', 'the sky is green', 'the dog is black']})

frame
                     a
0  Cat Mr. Nibbles is blue
1         the sky is green
2         the dog is black

pattern = '|'.join([f'(?i){animal}' for animal in mylist])  # python 3.6+

pattern
'(?i)dog|(?i)cat|(?i)fish'
 
frame.a.str.contains(pattern)
0     True  # Because of the (?i) flag, 'Cat' is also matched to 'cat'
1    False
2     True

edited May 11, 2021 at 0:20

Janosh

4,9973 gold badges37 silver badges39 bronze badges

answered Jul 31, 2013 at 14:31

Andy Hayden

378k110 gold badges640 silver badges546 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

zelusp Over a year ago

This significantly speeds up what I was doing. Is there any way to return the sub pattern (say, dog) matched instead of True False?

zelusp Over a year ago

Figured it out: to return the matched pattern use frame.a.str.extract(pattern)

Yog Over a year ago

@Andy Hayden How to print pattern values in case output is True

Yog Over a year ago

@Andy Hayden not it doesn't work tried it gives value error. Can you suggest something else.

Yog Over a year ago

I mean the expected output should be True Cat etc.., in place of True alone

|

Paul P · Accepted Answer · 2021-11-18 20:03:12Z

61

For list should work

print(frame[frame["a"].isin(mylist)])

See pandas.DataFrame.isin().

edited Nov 18, 2021 at 20:03

Paul P

4,0372 gold badges15 silver badges28 bronze badges

answered Feb 9, 2015 at 14:11

Meloun

15.3k17 gold badges70 silver badges94 bronze badges

2 Comments

HaPsantran Over a year ago

Will this work even if you're looking for potentially a substring from a list? That is, if you want to match any substring of column 'a' to any element in mylist, will this catch it?

Goutham Anush Over a year ago

No it does not work for substrings. It only matches the entire string and is case-sensitive.

Aman Raparia · Accepted Answer · 2020-02-27 07:56:32Z

9

After going through the comments of the accepted answer of extracting the string, this approach can also be tried.

frame = pd.DataFrame({'a' : ['the cat is blue', 'the sky is green', 'the dog is black']})

frame
              a
0   the cat is blue
1  the sky is green
2  the dog is black

Let us create our list which will have strings that needs to be matched and extracted.

mylist = ['dog', 'cat', 'fish']
pattern = '|'.join(mylist)

Now let create a function which will be responsible to find and extract the substring.

import re
def pattern_searcher(search_str:str, search_list:str):

    search_obj = re.search(search_list, search_str)
    if search_obj :
        return_str = search_str[search_obj.start(): search_obj.end()]
    else:
        return_str = 'NA'
    return return_str

We will use this function with pandas.DataFrame.apply

frame['matched_str'] = frame['a'].apply(lambda x: pattern_searcher(search_str=x, search_list=pattern))

Result :

              a             matched_str
   0   the cat is blue         cat
   1  the sky is green         NA
   2  the dog is black         dog

answered Feb 27, 2020 at 7:56

Aman Raparia

5031 gold badge5 silver badges14 bronze badges

2 Comments

Peter Chen Over a year ago

pattern_searcher() will also return those characters in the string. for example, catastrophe will also return cat. hotdog will return dog

Peter Chen Over a year ago

Moreover, if mylist = ['dog', 'cat', 'fish', dogs], the function will identify by order. for example, dogs are cool will return dog instead of dogs.

Ayesha Khan · Accepted Answer · 2020-11-01 14:19:48Z

0

We can check for three patterns simultaneously using pipe, for example

for i in range(len(df)):
       if re.findall(r'car|oxide|gen', df.iat[i,1]):
           df.iat[i,2]='Yes'
       else:
           df.iat[i,2]='No'

answered Nov 1, 2020 at 14:19

Ayesha Khan

1163 silver badges9 bronze badges

Collectives™ on Stack Overflow

Check if a string in a Pandas DataFrame column is in a list of strings

4 Answers 4

9 Comments

2 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

9 Comments

2 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related