Modifying multiple .csv files from same directory in python

Question

I need to modify multiple .csv files in my directory. Is it possible to do it with a simple script? My .csv columns are in this order:

 X_center,Y_center,X_Area,Y_Area,Classification

I would like to change them to this order:

 Classification,X_center,Y_center,X_Area,Y_Area

So far I managed to write:

import os
import csv

for file in os.listdir("."):
    if file.endswith(".csv"):
        with open('*.csv', 'r') as infile, open('reordered.csv', 'a') as outfile:
            fieldnames = ['Classification','X_center','Y_center','X_Area','Y_Area']
            writer = csv.DictWriter(outfile, fieldnames=fieldnames)
            writer.writeheader()
            for row in csv.DictReader(infile):
                writer.writerow(row)
        csv_file.close()

But it changes every row to Classification,X_center,Y_center,X_Area,Y_Area (replaces values in every row). Is it possible to open a file, re-order the columns and save the file under the same name? I checked similar solutions that were given on other threads but no luck. Thanks for the help!

are you okay with using pandas ? pip install pandas if so i'll write a solution below : — Umar.H
– Umar.H, Commented Dec 11, 2019 at 19:54

biomiker · Accepted Answer · 2019-12-11 20:23:19Z

First off, I think your problem lay in opening '*.csv' in the loop instead of opening file. Also though, I would recommend never overwriting your original input files. It's much safer to write copies to a new directory. Here's a modified version of your script which does that.

import os
import csv
import argparse

ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", required=True)
ap.add_argument("-o", "--output", required=True)
args = vars(ap.parse_args())


if os.path.exists(args["output"]) and os.path.isdir(args["output"]):
        print("Writing to {}".format(args["output"]))
else:
        print("Cannot write to directory {}".format(args["output"]))
        exit()

for file in os.listdir(args["input"]):
    if file.endswith(".csv"):
        print("{} ...".format(file))
        with open(os.path.join(args["input"],file), 'r') as infile, open(os.path.join(args["output"], file), 'w') as outfile:
            fieldnames = ['Classification','X_center','Y_center','X_Area','Y_Area']
            writer = csv.DictWriter(outfile, fieldnames=fieldnames)
            writer.writeheader()
            for row in csv.DictReader(infile):
                writer.writerow(row)
        outfile.close()

To use it, create a new directory for your outputs and then run like so:

python this.py -i input_dir -o output_dir

Note: From your question you seemed to want each file to be modified in place so this does basically that (outputs a file of the same name, just in a different directory) but leaves your inputs unharmed. If you actually wanted all the files reordered into a single file as your code open('reordered.csv', 'a') implies, you could easily do that by moving the output initialization code so it is executed before entering the loop.

Umar.H · Accepted Answer · 2019-12-11 19:59:22Z

1

Using pandas & pathlib.

from pathlib import Path # available in python 3.4 + 
import pandas as pd
dir = r'c:\path\to\csvs' # raw string for windows.
csv_files = [f for f in Path(dir).glob('*.csv')] # finds all csvs in your folder.


cols = ['Classification','X_center','Y_center','X_Area','Y_Area']

for csv in csv_files: #iterate list
    df = pd.read_csv(csv) #read csv
    df[cols].to_csv(csv.name,index=False)
    print(f'{csv.name} saved.')

naturally, if there a csv without those columns then this code will fail, you can add a try/except if that's the case.

answered Dec 11, 2019 at 19:59

Umar.H

23.1k7 gold badges50 silver badges94 bronze badges

Collectives™ on Stack Overflow

Modifying multiple .csv files from same directory in python

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related