Searching for strings in lists inside Pandas DataFrame

Question

I'm trying to search for strings within lists that are contained in a pandas dataframe, see this one example:

       userAuthor     hashtagsMessage
post_1    nytimes            [#Emmys]
post_2        TMZ                  []
post_3     Forbes        [#BTSatUNGA]
post_4    nytimes            [#Emmys]
post_5     Forbes  [#BTS, #BTSatUNGA]

As you have noticed, the column that hosts such lists is 'hashtagsMessage'. I've tried using conventional methods for string searching but I've not been able to.

If I wanted to look for an exact match for '#BTS', with a conventional method, you could use some of these options, like:

df['hashtagsMessage'].str.contains("#BTS", case=False)

or

df['hashtagsMessage']=="#BTS"

Or similar. Unfortunately, these approaches do not work for lists, I have to make an extra step I suppose to index inside the list while I'm searching in the DataFrame but I'm not really sure how to do this part.

Any help is entirely appreciated!

Do you search for the tag '#BTS' or a partial tag and match '#BTSatUNGA'? — Corralien
– Corralien, Commented Nov 25, 2021 at 13:18
@Corralien hi, forgot to add. Exact matches. Question was edited, thank you! — Aquiles Páez
– Aquiles Páez, Commented Nov 25, 2021 at 13:20

Corralien · Accepted Answer · 2021-11-25 13:23:50Z

2

Use map or apply:

>>> df['hashtagsMessage'].map(lambda x: '#BTS' in x)

post_1    False
post_2    False
post_3    False
post_4    False
post_5     True
Name: hashtagsMessage, dtype: bool

Update

A more vectorizable way using explode:

>>> df.loc[df['hashtagsMessage'].explode().eq('#BTS').loc[lambda x: x].index]

       userAuthor     hashtagsMessage
post_5     Forbes  [#BTS, #BTSatUNGA]

edited Nov 25, 2021 at 13:23

answered Nov 25, 2021 at 13:16

Corralien

121k8 gold badges43 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Corralien Over a year ago

I search for the tag '#BTS' and not a partial match.

wwnde · Accepted Answer · 2021-11-25 13:19:25Z

1

Please search for raw string

if not actual list use:

df['hashtagsMessage'].str.contains(r'#BTS')

if list please use

df['hashtagsMessage'].astype(str).str.contains(r'#BTS')

edited Nov 25, 2021 at 13:19

answered Nov 25, 2021 at 13:13

wwnde

26.7k6 gold badges22 silver badges38 bronze badges

Comments

mechanical_meat · Accepted Answer · 2021-11-25 13:20:59Z

1

You could use a simple anonymous function employing a list-comprehension and any() e.g.:

Edit: I originally presumed you wanted any tag containing '#BTS', and just edited to find only exact match(es) :)

In [10]: df = pd.DataFrame({'hashtagsMessage':[
                            [], ["#BTSatUNGA"],
                            ["#Emmys"], ['#BTS', '#BTSatUNGA']]})

In [18]: df['hashtagsMessage'].apply(lambda lst: any(s for s in lst
                                                     if s == "#BTS"))
Out[18]: 
0    False
1    False
2    False
3     True
Name: hashtagsMessage, dtype: bool

edited Nov 25, 2021 at 13:20

answered Nov 25, 2021 at 13:11

mechanical_meat

170k25 gold badges237 silver badges231 bronze badges

2 Comments

Aquiles Páez Over a year ago

Thanks for this, I forgot to be more clear in the original question, but your approach I also liked very much because it will have other uses cases (similar actually) where matching doesn't need to be exact.

mechanical_meat Over a year ago

@AquilesPáez: ok! happy coding to you :)

Collectives™ on Stack Overflow

Searching for strings in lists inside Pandas DataFrame

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related