open a CSV file once and write several lines to it from a loop python

Question

The following method:

def generateCSVfile(fileName,fileDescription,fileLocation,md5Hash):
    with open('deploymentTemplate.csv', 'w') as csvfile:
        createRow = csv.writer(csvfile,
                                 quoting=csv.QUOTE_MINIMAL)

This generates my CSV file but since I am calling it in a loop it just overrides itself.

generateCSVfile(name, fileDescription, filePath+"/"+name, md5Hash)

I am trying to find a way to generate the file, leave it open, call the above method and have all the text written to it without the file overriding itself.

Community · Accepted Answer · 2020-06-20 09:12:55Z

1

Use : open('deploymentTemplate.csv', 'a') to append values.

Syntax: open(<file_name> [,<mode>])

Different modes are :

mode can be 'r' when the file will only be read
'w' for only writing (an existing file with the same name will be erased)
'a' opens the file for appending and any data written to the file is automatically added to the end.
'r+' opens the file for both reading and writing.

The mode argument is optional; 'r' will be assumed if it’s omitted.

Eg :

with open("test.txt", "a") as myfile:
    myfile.write("appended text")

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered May 19, 2016 at 18:28

Ani Menon

28.4k17 gold badges111 silver badges128 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Mike3355 Over a year ago

Yes I thought of that but this file may be run a few times a week. I suppose I could clear the file when the project first runs then append.

Ani Menon Over a year ago

Open the file 'w' mode, do an empty write & then open in 'a' mode. OR you may do this file.seek(0);file.truncate() (to clear file contents).

ShadowRanger · Accepted Answer · 2016-05-19 20:59:46Z

If the file needs to be emptied once per program run, but appended multiple times within a run, you could always just use a global (or class member state) to ensure it's only opened once.

import atexit

csvfile = None
def generateCSVfile(fileName,fileDescription,fileLocation,md5Hash):
    global csvfile
    if csvfile is None:
        # Lazily open file on first call
        csvfile = open('deploymentTemplate.csv', 'w')
        atexit.atexit(csvfile.close)  # Close cleanly on program exit

    try:
        csvwriter = csv.writer(csvfile, quoting=csv.QUOTE_MINIMAL, newline='')
        # do whatever writing you need to csvwriter
    finally:
        csvfile.flush()  # Match behavior of repeated with/open, force predictable flush

If there might be multiple CSV files involved, you might use a class with instance state and a method to do the writing, so each file can be independently cleared once and appended many times. In this case, due to limits on the number of open file handles, reopening for append on each use is slower but safer than opening once and leaving open. You can use caching so the class is a singleton for any given file name too:

import weakref

class CSVGenerator:
    CACHE = {}
    CACHELOCK = threading.Lock()

    def __new__(cls, csvfilename):
        canonicalname = os.path.realpath(csvfilename)
        newself = super().__new__(cls)
        with cls.CACHELOCK:
            self = cls.CACHE.setdefault(canonicalname, newself)
            if newself is self:
                # First time we opened this file, clear file and initialize instance
                with open(canonicalname, 'w') as f:
                    pass
                self.csvfilename = canonicalname
                self.appendlock = threading.Lock()
        return self

    def generateCSVfile(self, fileName, fileDescription, fileLocation, md5Hash):
        with newself.appendlock, open(self.csvfilename, 'a', newline='') as csvfile:
            createRow = csv.writer(csvfile, quoting=csv.QUOTE_MINIMAL)
            # Perform writes to file

Usage of the class can be either:

 CSVGenerator(somecsvfilename).generateCSVfile(...args...)

which acquires an instance briefly (creating it if needed) then writes once, or it can create and store an instance and reuse it (saves the overhead of cache lookup, but functionally identical).

Collectives™ on Stack Overflow

open a CSV file once and write several lines to it from a loop python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related