Dynamically read and load files into Python

Question

In Python, is there a way to import csv or text files dynamically.We process multiple files a week that have different names and I don't want to update the with open statement manually each time the script runs. I have a function to read the file name which I pass to a variable for later use in my code.

I can see and read the files in the directory but I am not sure if I can add the contents of the folder into a variable that can then be used in the with open statement.

import os
os.chdir('T:\Credit Suite')
DIR = os.listdir()
print(DIR)

import csv,sys
with open('July 19.csv',mode='r') as csv_file:
    ROWCOUNT = 0
    FILENAME = (csv_file.name)
    output = csv.writer(open('test2.txt', 'w', newline=''))
    reader =csv.DictReader(csv_file)
    for records in reader:
        ROWCOUNT += 1
        EIN = records['EIN']
        DATE = records['Date Established']
        DUNS = records['DUNS #']
        COMPANYNAME = records['Company Name']
        lineout =('<S>'+ EIN+'$EIN '+EIN+'*'+DATE+')'+ COMPANYNAME +'#D-U-N-S '+DUNS).upper()
        output.writerow([lineout])
        print("writing completed")

I will be running my script when a file hits a folder using a monitor and scheduler in an automated process. I want the code to run no matter what the inbound file name is labeled as in the folder and I wont have to update the code manually for the file name or change the file name to a standard name each time.

sstibal · Accepted Answer · 2019-08-02 22:34:56Z

1

os.chdir('T:\Credit Suite')
for root, dirs, files in os.walk("."):
    for filename in files:
        if filename.endswith('.csv'):
            f=filename
import csv,sys
with open(f,mode='r') as csv_file:

answered Aug 2, 2019 at 22:34

sstibal

231 silver badge6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ran · Accepted Answer · 2019-08-04 07:41:01Z

1

os.listdir() returns a list of all the files in the dir, you can just loop all the files:

import os
os.chdir('T:\Credit Suite')
DIR = os.listdir()
print(DIR)

import csv,sys
for file in DIR:
    if file.endswith('.csv'):
        with open(file,mode='r') as csv_file:
            ROWCOUNT = 0
            FILENAME = (csv_file.name)
            output = csv.writer(open(FILENAME + '_output.txt', 'w', newline=''))
            reader =csv.DictReader(csv_file)
            all_lines = []
            for records in reader:
                ROWCOUNT += 1
                EIN = records['EIN']
                DATE = records['Date Established']
                DUNS = records['DUNS #']
                COMPANYNAME = records['Company Name']
                lineout =('<S>'+ EIN+'$EIN '+EIN+'*'+DATE+')'+ COMPANYNAME +'#D-U-N-S '+DUNS).upper()
                all_lines.append(lineout)
            output.writerow(all_lines)
            print("writing completed")
        # remove file to avoid reprocessing the file again in the next run
        # of the script, or just move it elsewhere with os.rename
        os.remove(file)

edited Aug 4, 2019 at 7:41

answered Aug 2, 2019 at 22:11

ran

211 silver badge4 bronze badges

2 Comments

sstibal Over a year ago

Thank you. As soon as I posted this question I found the os.walk method as well. I posted the code below that I currently have and its working as well. Not sure if its the best way to do it. I will give your code a try as well.

ran Over a year ago

keep in mind that os.walk search all the hierarchy of the folder as appose to os.listdir that just list the current dir content. also in your code you will only process the last csv file that os.walk will find because it reassigning the variable f. not sure if that what you wanted or not. I also edited the answer to include variable output file names so it will generate output file for each processed csv

Collectives™ on Stack Overflow

Dynamically read and load files into Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related