How to open and iterate through a list of CSV files - Python

Question

I have a list of pathnames to CSV files. I need to open each CSV file, take the data without the header and then merge it all together into a new CSV file.

I have this code which gets me the list of CSV file pathnames:

file_list = []
for folder_name, sub_folders, file_names in os.walk(wd):
    for file_name in file_names:
        file_extention = folder_name + '\\' + file_name
        if file_name.endswith('csv'):
            file_list.append(file_extention)

An example of my list is:

['C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data1.csv',
 'C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data2.csv',
 'C:\\Users\\Documents\\GPS_data\\West_coast\\Westland\\GPS_data3.csv']

I am struggling to figure out what to do, any help would be greatly appreciated. Thanks.

Seams like you need to look up how to open and read a file in Python, this is well documented: docs.python.org/3/library/filesys.html — Apeiron
– Apeiron, Commented Mar 16, 2018 at 3:06
You should not write that nested loop. docs.python.org/3/library/glob.html — OneCricketeer
– OneCricketeer, Commented Mar 16, 2018 at 3:34

Rusty · Accepted Answer · 2018-03-16 18:03:31Z

2

The main idea is to read in each line of a file, and write it to the new file. But remember to skip the first line that has the column headers in it. I previously recommend the cvs module, however it doesn't seem like that is necessary, since this task does not require analyzing the data.

file_list = ['data1.csv','data2.csv']

with open('new.csv', 'w') as newfile:  # create a new file
    for filename in filelist:
        with open(filename) as csvfile:
            next(csvfile)   # skip the header row
            for row in csvfile:
                newfile.write(line) # write to the new csv file

Edit: clarified my answer.

edited Mar 16, 2018 at 18:03

answered Mar 16, 2018 at 3:16

Rusty

1,14310 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

OneCricketeer Over a year ago

The question seems to be asking about merging files. You're only building a list

kels Over a year ago

What a legend! This is exactly what I've been trying to do, thanks for the help :)

Collectives™ on Stack Overflow

How to open and iterate through a list of CSV files - Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related