Pandas dataframe: converting column of lists to a list

Question

I have a dataframe df with a column hashtags such that:

df['hashtags']
>>>
0                                                       NaN
1                                                       NaN
2                                               ['COVID19']
3                                               ['COVID19']
4                         ['CoronaVirusUpdates', 'COVID19']
                                ...                        
132596    ['coronacrise', 'covid19', 'JN', 'NãoÉSóUmNúme...
132597                                          ['covid19']
132598                                ['corona', 'covid19']
132599                                                  NaN
132600                                          ['covid19']
Name: hashtags, Length: 132601, dtype: object

I want to create a list containing all the lists' elements (except the Nan) of the column.
I have tried to make a list of lists by:

li = df['hashtags'].tolist()

But it's converting the lists into a string and end up with a list of strings. For example:

li[:5]
>>> 
[nan, nan, "['COVID19']", "['COVID19']", "['CoronaVirusUpdates', 'COVID19']"]

My desired output for li[:5] is like:

['COVID19', 'COVID19', 'CoronaVirusUpdates', 'COVID19', 'coronavirus', 'covid19']

@MichaelO. I need to make a list of lists first, that's where I am facing issue. — raf
– raf, Commented Aug 15, 2020 at 7:32

jezrael · Accepted Answer · 2020-08-15 07:35:03Z

2

Idea is first remove missing values by Series.dropna, then convert list repr by ast.literal_eval to lists and flatten nested lists in list comprehension:

df = pd.DataFrame({'hashtags':[np.nan, np.nan, 
                               "['COVID19']", "['COVID19']", 
                               "['CoronaVirusUpdates', 'COVID19']"]})

import ast

out = [y for x in df['hashtags'].dropna() for y in ast.literal_eval(x)]
print (out)
['COVID19', 'COVID19', 'CoronaVirusUpdates', 'COVID19']

edited Aug 15, 2020 at 7:35

answered Aug 15, 2020 at 7:27

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pandas dataframe: converting column of lists to a list

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related