Pandas combine multiple csv files

Question

I have multiple csv files that I would like to combine into one df.

They are all in this general format, with two index columns:

                                           1     2
CU0112-005287-7 Output Energy, (Wh/h)   0.064   0.066
CU0112-005287-7 Lights (Wh)                0     0

                                            1     2
CU0112-001885-L Output Energy, (Wh/h)   1.33    1.317
CU0112-001885-L Lights (Wh)             1.33    1.317

and so on...

The combined df would be:

                                           1     2
CU0112-005287-7 Output Energy, (Wh/h)   0.064   0.066
CU0112-005287-7 Lights (Wh)                0     0
CU0112-001885-L Output Energy, (Wh/h)   1.33    1.317
CU0112-001885-L Lights (Wh)             1.33    1.317

I am trying this code:

import os
import pandas as pd
import glob

files = glob.glob(r'2017-12-05\Aggregated\*.csv')   //folder which contains all the csv files

df = pd.merge([pd.read_csv(f, index_col=[0,1])for f in files], how='outer')

df.to_csv(r'\merged.csv')

But I am getting this error:

TypeError: merge() takes at least 2 arguments (2 given)

jezrael · Accepted Answer · 2018-03-05 13:02:27Z

4

I think you need concat instead merge:

df = pd.concat([pd.read_csv(f, index_col=[0,1]) for f in files])

answered Mar 5, 2018 at 13:02

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Yayati Sule · Accepted Answer · 2018-03-05 13:08:18Z

0

You can try the following. I made some changes to the DataFrame combining logic

import os
import pandas as pd
import glob

files = glob.glob(r'2017-12-05\Aggregated\*.csv')   //folder which contains all the csv files

df = reduce(lambda df1,df2: pd.merge(df1,df2,on='id',how='outer'),[pd.read_csv(f, index_col=[0,1])for f in files] )

df.to_csv(r'\merged.csv')

answered Mar 5, 2018 at 13:08

Yayati Sule

1,62914 silver badges25 bronze badges

Comments

Billy Bonaros · Accepted Answer · 2018-03-05 15:30:42Z

0

A simple way:

Creating a list with the names of csvs:

files=listdir()
csvs=list()
for file in files:
    if file.endswith(".csv"):
        csvs.append(file)

concatenate the csvs:

data=pd.DataFrame()
for i in csvs:
    table=pd.read_csv(i)
    data=pd.concat([data,table])

answered Mar 5, 2018 at 15:30

Billy Bonaros

1,73114 silver badges19 bronze badges

Collectives™ on Stack Overflow

Pandas combine multiple csv files

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related