Iterating a user defined function over a list of dataframes in python

Question

I have two really large dataframes that I'd like to merge, but can't because my computer's memory cannot handle it. Instead, I would like to split one of the dataframes into smaller dataframes by group (of which there are 195 groups, so 195 dataframes), and then join each of those 195 dataframes to the other large dataframe.

So far I have tried groupby:

split_data = list(bigdata1.groupby("GROUP"))

Which results in a list of the 195 dataframes

I would now like to know how to apply the joining function defined below to each of the dataframes, and have each of the dataframes separated and able to be called for subsequent manipulation (e.g. analysis). I am brand new to python so any assistance would be greatly appreciated. Thanks in advance

def joining_function(df):
    pd.merge(df, bigdata2, on = 'PERSON_ID', how = 'left')

Ben.T · Accepted Answer · 2021-07-15 00:09:26Z

1

First your function need to return the result:

def joining_function(df):
    return pd.merge(df, bigdata2, on = 'PERSON_ID', how = 'left')

Then a better way to store all the dataframes is in a dictionary using the group name as a key. Use the function at this moment

res = {gr:joining_function(dfg) for gr, dfg in bigdata1.groupby("GROUP")}

answered Jul 15, 2021 at 0:09

Ben.T

29.7k6 gold badges39 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Iterating a user defined function over a list of dataframes in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related