Concatenate multiple similar CSV files into one big dataframe

Question

I have one directory where there are only the CSV files I want to use. I want to concatenate all these CSV files and create a bigger one. I've tried one code but it didn't work.

import os
import pandas as pd

targetdir = r'C:/Users/toshiba/Documents/ICF2011/Base Admision San Marcos 2014-2/Sabado'

filelist = os.listdir(targetdir) 

big_df=pd.DataFrame()
for file in filelist :
    big_df.append(pd.read_csv(file), ignore_index=True)

I run the code and a there's a message that says: IOError: File A011.csv does not exist. This is contradictory because A011.csv is the first CSV file in the directory that I used.

chthonicdaemon · Accepted Answer · 2014-03-11 06:56:40Z

2

listdir only returns the filename, not the complete path. To get the complete path you will need to join targetdir and file (bad variable name as it masks the file type). Also, you will have to capture the result of .append as it returns a new object rather than appending in place.

for filename in filelist:
    big_df = big_df.append(pd.read_csv(os.path.join(targetdir, filename), ignore_index=True)

answered Mar 11, 2014 at 6:56

chthonicdaemon

19.9k2 gold badges55 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Andy Hayden · Accepted Answer · 2014-03-11 07:22:49Z

2

As mentioned in the other answer, you need to use the full-path, rather than the local path.

I recommend using concat rather than append, as this way you don't make many intermediary frames:

big_df = pd.concat(pd.read_csv(os.path.join(targetdir, filename),
                               ignore_index=True)
                   for filename in filelist)

answered Mar 11, 2014 at 7:22

Andy Hayden

378k110 gold badges640 silver badges546 bronze badges

Collectives™ on Stack Overflow

Concatenate multiple similar CSV files into one big dataframe

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related