Faster way of creating List of Lists from Pandas Dataframe

Question

I have a large pandas data frame with the following format

DATE         ID    ACTION
01/12/2014    1       A
01/12/2014    1       B
02/12/2014    1       C
02/12/2014    1       D
01/12/2014    2       E
02/12/2014    2       F
02/12/2014    2       E
04/12/2014    2       G

Can create the data as follows:

import pandas as pd

df = pd.DataFrame({'DATE': ['01/12/2014','01/12/2014','02/12/2014','01/12/2014','02/12/2014','02/12/2014','02/12/2014','04/12/2014' ],
                   'ID': [1,1,1,1,2,2,2,2],
                   'ACTION': ['A', 'B', 'C', 'D', 'E', 'F', 'E', 'G']})

From this I want to create a list of lists for each Date/ID Grouping. At the moment here's what I'm doing... it works, but I have millions of rows so it takes hours to run. Are there any more efficient ways to achieve the same result?

listoflists = [group['ACTION'].str.strip().tolist() for name, group in df.groupby(level=['DATE', 'ID'])]

Output:

[['A', 'B', 'D'], ['C'], ['E', 'F', 'E'], ['G']]

Since Python does not know the size of the final list in advance, as the list grows Python may be forced to allocate more space, and copy the list many times. That can make building the list quite slow. What do you want to do with the list once it is formed? Perhaps there is a way to achieve that goal without building the list of lists. — unutbu
– unutbu, Commented Dec 15, 2014 at 14:50
I use the listoflists in a word2vec model (radimrehurek.com/gensim/models/word2vec.html). I did see something about streaming from a file, but my file is in the table format and needs to be transformed first. — Ger
– Ger, Commented Dec 15, 2014 at 15:26

unutbu · Accepted Answer · 2014-12-15 20:07:15Z

1

Accord to this tutorial:

Gensim only requires that the input must provide sentences sequentially, when iterated over. No need to keep everything in RAM: we can provide one sentence, process it, forget it, load another sentence…

Therefore, you could use a memory-efficient generator expression instead of a list comprehension:

sentences = (group['ACTION'].str.strip().tolist() 
             for name, group in df.groupby(level=['DATE', 'ID']))

model = gensim.models.Word2Vec(sentences, ...)

answered Dec 15, 2014 at 20:07

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ger Over a year ago

Thanks for the suggestion, but unfortunately it didn't work for me. I'm not sure the exact issue but it seemed to have memory problems. Just a thought, but it seems Word2Vec makes two passes over the sentences object, once to create a vocabulary and once to train the model, so might that mean it has to evaluate the 'sentences' generator twice, whereas the other method takes the overhead of creating the list up front, speeding up the two passes? Not really sure though.

Collectives™ on Stack Overflow

Faster way of creating List of Lists from Pandas Dataframe

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related