Writing csv with quotes around strings (Python)

Question

I have written the following code to take a large csv file, and split it into multiple csv files based on a particular word in a column. The original csv file has some fields that are strings, and they have quotes around them.

For example:

Field1,Field2,Field3,Field4
1,2,"red",3
1,4,"red",4
3,4,"blue",4

etc.

My code splits the file into separate csvs based on Field4.

My output looks like this:

3.csv
Field1,Field2,Field3,Field4
1,2,red,3

4.csv
Field1,Field2,Field3,Field4
1,4,red,4
3,4,blue,4

I want my output to maintain quotes around the strings in field 3. The files are fed into a piece of software that only works if strings have quotes around them, which is quite annoying.

My current code looks like this:

import csv

#Creates empty set - this will be used to store the values that have already been used
newfilelist = set()

#Opens the large csv file in "read" mode
with open('File.csv', 'r') as csvfile:
    
    #Read the first row of the large file and store the whole row as a string (headerstring)
    read_rows = csv.reader(csvfile)
    headerrow = next(read_rows)
    headerstring=','.join(headerrow) 
    for row in read_rows:
        
        #Store the whole row as a string (rowstring)
        rowstring=','.join(row)
        
        #Takes Field 4
        newfilename = (row[3])
        
        
        #This basically makes sure it is not looking at the header row.
        if newfilename != "field4":
        
            
            #If the newfilename is not in the newfilename set, add it to the list and create new csv file with header row.
            if newfilename not in newfilelist:    
                newfilelist.add(newfilename)
                with open('//output/' +str(newfilename)+'.csv','a') as f:
                    f.write(headerstring)
                    f.write("\n")
                    f.close()    
            #If the newfilename is in the newfilelist set, append the current row to the existing csv file.     
            else:
                with open('//output/' +str(newfilename)+'.csv','a') as f:
                    f.write(rowstring)
                    f.write("\n")
                    f.close()

Can anybody advise me how to get the quotes around the strings? Unfortunately the software that uses my files requires them to be in this format!

Probably a dup of this stackoverflow.com/questions/36628847/… — Andrey Belykh
– Andrey Belykh, Commented Nov 8, 2017 at 19:17

Ignacio Vazquez-Abrams · Accepted Answer · 2017-11-08 19:15:14Z

17

Pass quoting=csv.QUOTE_NONNUMERIC to csv.writer().

answered Nov 8, 2017 at 19:15

Ignacio Vazquez-Abrams

804k160 gold badges1.4k silver badges1.4k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

WombatPM · Accepted Answer · 2017-11-08 20:46:16Z

0

CSVwriter might be overkill for what you are trying to do. If you want the entire line unchanged, just write the entire line.

#Creates empty array - this will be used to store the values that have already been used
newfilelist = {}

#Opens the large csv file in "read" mode
with open('File.csv, 'r') as csvfile:

    #Read the first row of the large file and store the whole row as a string (headerstring)
    headerstring = csvfile.readline()
    for row in csvfile.readlines():

        #Takes Field 4
        newfilename = row.split(',')[3].strip('"')

        #If the newfilename is not in the newfilename set, add it to the list and create new csv file with header row.
        if newfilename not in newfilelist:    
            newfilelist[newfilename] = open('//output/' +str(newfilename)+'.csv','w'):  #open a file and store the file reference in an dictionary
            newfilelist[newfilename].write(headerstring)

        newfilelist[newfilename].write(row)  # Write out a row to an existing file

#Close all open files
for k in newfilelist.keys():
    newfilelist[k].close()

answered Nov 8, 2017 at 20:46

WombatPM

2,6392 gold badges23 silver badges24 bronze badges

3 Comments

Actuary Over a year ago

This code seems to break if file sizes are huge, but the initial code works. I get MemoryError at the "for row in csvfile.readlines():" line.

WombatPM Over a year ago

Not suprising if you your output logic generates a lot of files. If you know a point where you will no longer be writing to a file, you can close earlier instead of waiting to the end.

Actuary Over a year ago

I guess to do this I'd have to sort the data? There is not necessarily any order and there could be tens of thousands of files. I think sorting the data would be tricky with the initial file sizes being multiple gigabytes!

Collectives™ on Stack Overflow

Writing csv with quotes around strings (Python)

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related