Fast and Pythonic way to turn nested list into pd Dataframe

Question

I have a list of which each element is a list with strings from a book

test_list = [['I love Stackoverflow', 'For ever', 'and always'], ['I dont like rain', 'it is wet']]
book_names = ['message to SO', 'confessions']

I would like to obtain the following dataframe


          book              sentence
0  message to SO  I love Stackoverflow
1  message to SO              For ever
2  message to SO            and always
3    confessions      I dont like rain
4    confessions             it is wet

Now, I managed to do this with the following piece of code:

df = pd.DataFrame(test_list, index=book_names).stack().reset_index(level=0)
df.rename(columns={'level_0':'book',
                    0 : 'sentence'},
                    inplace = True)

Resulting in :

            book              sentence
0  message to SO  I love Stackoverflow
1  message to SO              For ever
2  message to SO            and always
0    confessions      I dont like rain
1    confessions             it is wet

Now i have to reindex the result:

df.reset_index(drop=True)

I am not particularly happy with this code, having to reset_index and renaming columns. Anyone has a better solution?

In reality the test_list is rather large so speed is also an important consideration

Thanks in advance

jezrael · Accepted Answer · 2021-03-18 09:15:54Z

3

I think here is best create list of tuples in list comprehension with zip and pass to DataFrame constructor:

df = pd.DataFrame([(b,s) for b, n in zip(book_names, test_list) for s in n], 
                   columns=['book','sentence'])
print (df)
            book              sentence
0  message to SO  I love Stackoverflow
1  message to SO              For ever
2  message to SO            and always
3    confessions      I dont like rain
4    confessions             it is wet

Only pandas solution is with DataFrame.explode:

df = pd.DataFrame({'book':book_names ,
                   'sentence':test_list}).explode('sentence').reset_index(drop=True)
print (df)
            book              sentence
0  message to SO  I love Stackoverflow
1  message to SO              For ever
2  message to SO            and always
3    confessions      I dont like rain
4    confessions             it is wet

answered Mar 18, 2021 at 9:15

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Rens Over a year ago

Thanks a lot, amazing answer! One more question, why would you prefer the tuple/listcomprehension approach over the pandas solution? -- EDIT: nm, I prefer it myself as well, much cleaner. Thanks

jezrael Over a year ago

@Rens - Hmm, I think it depends of data, but I think list comprehension should be faster here. Th best test performance in real data

Collectives™ on Stack Overflow

Fast and Pythonic way to turn nested list into pd Dataframe

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related