Merge a list of list of dataframe python

Question

I have a list of lists each containing a common column, A, B to be used as indices as follows

df_list=[[df1], [df2],[df3], [df4], [df5, df6]]

I would like to merge the dataframes into a single dataframe based on the common columns A, B in all dataframes

I have tried pd.concat(df_list). This doesnt work and yields an error

 list indices must be integers or slices, not list

Is there a way to accomplish this

Yo need to be more specific. Why do you have now a list of lists in df_lists? — Maciałek
– Maciałek, Commented Sep 2, 2020 at 7:22
[d5, d6] is a list and not an integer, that is why you are getting that error. Can you elaborate more specifically like you have to create a column of lists or what? — user13897142
– user13897142, Commented Sep 2, 2020 at 7:25

Daweo · Accepted Answer · 2020-09-02 07:29:36Z

4

You need to deliver flat list (or other flat structure) to pd.concat your

df_list=[[df1], [df2],[df3], [df4], [df5, df6]]

is nested. If you do not have command over filling said df_list you need to flatten it first for example using itertools.chain as follow:

import itertools
flat_df_list = list(itertools.chain(*df_list))

and then deliver flat_df_list or harnessing fact that pd.concat also work with itertators:

total_df = pd.concat(itertools.chain(*df_list))

answered Sep 2, 2020 at 7:29

Daweo

38.2k3 gold badges17 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

snatchysquid · Accepted Answer · 2020-09-02 07:20:15Z

3

try simply using something like:

dfs = [df1, df2, df3, ... etc]

print(pd.concat(dfs))

when storing the dfs in a list you shouldn't keep them in a second list, note pd.concat takes a list of dfs.

answered Sep 2, 2020 at 7:20

snatchysquid

1,38212 silver badges25 bronze badges

Raghavan vmvs · Accepted Answer · 2020-09-02 07:29:19Z

1

The solution is as follows

  flat_list = [item for sublist in df_list for item in sublist]
        
  #L3 = pd.DataFrame.from_dict(map(dict,D))
  L3=pd.concat(flat_list)

answered Sep 2, 2020 at 7:29

Raghavan vmvs

1,3051 gold badge12 silver badges32 bronze badges