Combine multiple CSV files using Python and Pandas

Question

I have the following code:

import glob
import pandas as pd
allFiles = glob.glob("C:\*.csv")
frame = pd.DataFrame()
list_ = []
for file_ in allFiles:
    print file_
    df = pd.read_csv(file_,index_col=None, header=0)
    list_.append(df)
    frame = pd.concat(list_, sort=False)
print list_
frame.to_csv("C:\f.csv")

This combines multiple CSVs to single CSV.

However it also adds a row number column.

Input:

a.csv

a   b   c   d
1   2   3   4

b.csv

a   b   c   d
551 55  55  55
551 55  55  55

result: f.csv

    a   b   c   d
0   1   2   3   4
0   551 55  55  55
1   551 55  55  55

How can I modify the code not to show the row numbers in the output file?

Bera · Accepted Answer · 2018-06-27 13:38:12Z

2

Change frame.to_csv("C:\f.csv") to frame.to_csv("C:\f.csv", index=False)

See: pandas.DataFrame.to_csv

answered Jun 27, 2018 at 13:38

Bera

2,2203 gold badges22 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

nosklo · Accepted Answer · 2018-06-27 13:48:11Z

1

You don't have to use pandas for this simple task. pandas is parsing the file and converting the data to numpy constructs, which you don't need... In fact you can do it with just normal text file manipulation:

import glob
allFiles = glob.glob("C:\*.csv")
first = True
with open('C:\f.csv', 'w') as fw:
    for filename in allFiles:
        print filename
        with open(filename, 'r') as f:
            if not first:
                f.readline() # skip header
            first = False
            fw.writelines(f)

edited Jun 27, 2018 at 13:48

answered Jun 27, 2018 at 13:41

nosklo

224k58 gold badges299 silver badges299 bronze badges

2 Comments

jack Over a year ago

This code also adds the header from each file. The header should be shown only once. The fw.writelines(f) needs to be conditional - write the row only if it's no the header row except the first time.

nosklo Over a year ago

I fixed it with a flag @jack

Collectives™ on Stack Overflow

Combine multiple CSV files using Python and Pandas

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related