Writing and appending multiple csv data into new csv using python

Question

I have a directory where there are multiple csv files. Currently I am able to read all the files sequentially using for loop and display their contents. I need to to write the contents from all the csv files sequentially into a new csv file but I am missing something as in my new csv has no data in it.

this is what I am doing :

import os
import csv     

path = r'C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ\\' 
fileNames = os.listdir(path)

for f in fileNames:
    file = open(path+f)
    csvreader = csv.reader(file)
    
    rows = []
    for row in csvreader:
        rows.append(row)
      
    for i in rows:
        print(i)
        #OFile = open('C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ\ALL_DATA.csv','w')
        writer = csv.writer(open('C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ\ALL_DATA.csv', 'wb'))
        #for row in csvreader:
         #   row1 = csvreader.next()
        writer.writerow(i)

chill0r · Accepted Answer · 2022-03-22 13:32:45Z

2

You are overwriting the file each row you try to write. Using the w argument for the open method will overwrite existing files.

The argument you need to use in the case you want to append to files (or create new files if non-existing) is a

See Python File Open for more informations about python file modes.

import os
import csv     

path = r'C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ' 
fileNames = os.listdir(path)

with open('C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ\ALL_DATA.csv', 'a') as output:    
    writer = csv.writer(output)
    for f in fileNames:
        with open(os.path.join(path, f), "r") as file:
            csvreader = csv.reader(file)
            for row in csvreader:
                print(row)
                writer.writerow(row)

If the csv files have the same columns and formats you could also simply copy the first file and append the others, excluding their headers.

import os
import shutil

path = r'C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ'
fileNames = os.listdir(path)
output = r'C:\Users\hu170f\Documents\WORK\MAAP_FILE_DB_REQ\ALL_DATA.csv'

# Copy the first file:
shutil.copyfile(os.path.join(path,fileNames[0]), output)

# Append the remaining file contents, excluding each first line
with open(output, 'a') as out:
    for file in fileNames[1:]:
        with open(os.path.join(path, file), 'r') as in_:
            out.write(''.join(in_.readlines()[1:]))

edited Mar 22, 2022 at 13:32

answered Mar 22, 2022 at 12:31

chill0r

1,1292 gold badges10 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

shikha Over a year ago

it is showing error for line "with open(os.path.join(path), file), 'r')" stating as invalid syntax and if I add opening braces before os.path then the error changes to No directory present for ALL_DATA

chill0r Over a year ago

@shikha Sorry, there was a typo. The ) after path needs to be removed. I edited the answer

shikha Over a year ago

getting error as No such file or directory: 'ALL_DATA.csv' even though the file exist.

chill0r Over a year ago

@shikha I changed the example to use an absolute path. I tested it while I was in the directory. shutil.copyfile takes either an absolute or relative path. Furthermore I found another problem in the out.write line while testing it.

Collectives™ on Stack Overflow

Writing and appending multiple csv data into new csv using python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related