Python Concatenate Multiple CSV files with no header

Question

So I have about 3,000 csv files and they are all named differently. Example, CDEE.csv and the structure is just one line containing the name and an amount.

CDEE | 3993

I tried to concatenate and I keep getting

CDEE | 3993 | AASE| 3939 .........

but I want

CDEE | 3992
AASE | 3939
xxxx | yyyy

Here is the code: import pandas as pd import glob, os

path = "/home/username/myfolder" 
os.chdir(path)
results = pd.DataFrame([])

for counter, file in enumerate(glob.glob(".csv*")):
    namedf = pd.read_csv(file,skiprows=0, usecols=[1,2,3])
    results = results.append(namedf)

results.to_csv('Combined.csv')

Thanks for any help, I really appreciate it!

Are you actually doing anything with the DataFrame or just purely wanting to concat the files? As cat *.csv > csv_files.all is looking pretty promising right now... — Jon Clements
– Jon Clements, Commented Aug 18, 2017 at 14:38

Mohamed Ali JAMAOUI · Accepted Answer · 2017-08-18 15:49:52Z

3

You need to use pd.concat which is documented here

import pandas as pd
import os
import glob

path = "."
os.chdir(path)
results = pd.DataFrame()

for counter, current_file in enumerate(glob.glob("*.csv")):
    namedf = pd.read_csv(current_file, header=None, sep="|")
    print(namedf)
    results = pd.concat([results, namedf])

results.to_csv('Combined.csv', index=None, header=None, sep="|")

Note that there are few mistakes to fix:

change glob.glob(".csv*") to glob.glob("*.csv") to get all files that end with .csv
to get exactly the following output:

CDEE|3992
AASE|3939
xxxx|yyyy

You need to call df.to_csv with index=None to not write the index, header=None to not write the header and sep="|" to use | as separator instead of the default ,

edited Aug 18, 2017 at 15:49

answered Aug 18, 2017 at 15:30

Mohamed Ali JAMAOUI

14.8k14 gold badges79 silver badges124 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Concatenate Multiple CSV files with no header

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related