How can I combine the dataframes returned by a function efficiently in python?

Question

I have a function which returns many values on every run but I am interested in dataframe being returned. I am currently using a list to store dataframes and then at the end I use pd.concat to convert all the dfs in to a single df. It's working fine but when I try to push this code for review I face linting errors "too many local variables'. There's no way I can deal with other varibales so, is there a way more efficient to store these dataframes and then later convert it to a single frame.

Currently, the code is something like this:

df_list = []

....

_, _, df = function_call()

df_list.append(df)

....

df = pd.concat(df_list, axis=0)

Where is stored the dataframes? csv, excel, html? Show us a part of your df and how much dfs and rows will be concat. — Wilian
– Wilian, Commented Dec 13, 2021 at 10:23
This dataframe is getting stored inside big query. The dfs could range from 10 to 100s to even 1000s. Even with the rows, could go into 100,000. — Budding_Data_Analyst
– Budding_Data_Analyst, Commented Dec 13, 2021 at 10:35

Arne · Accepted Answer · 2021-12-13 12:06:43Z

1

Would it be possible to store the argument values to pass to the dataframe generating function in a list, and then use a generator expression in the pd.concat call?

Something like this:

kwargs_list = []

kwargs = {'arg1': val1, 'arg2': val2}
kwargs_list.append(kwargs)
...

df = pd.concat(function_call(**kwargs) for kwargs in kwargs_list)

answered Dec 13, 2021 at 12:06

Arne

10.6k2 gold badges22 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Budding_Data_Analyst Over a year ago

Thanks for the suggestion but the function also return other values apart from the dataframe. I should have mentioned this in the question as well.

alparslan mimaroğlu Over a year ago

do you need the other values? If you don't need them adding a lambda function will solve the problem

Budding_Data_Analyst Over a year ago

Yes, other values are being used to achieve something else. I just need to find a better way of doing this, so that I can reduce the count of variables.

Collectives™ on Stack Overflow

How can I combine the dataframes returned by a function efficiently in python?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related