How to load multiple csv files to an iterable variable in Python?

Question

I want to open a variable number of csv files and then I would like to iterate over the csv files opened and upload 1 row of each file at a time to my sql database. For example, loop through each file uploading the first row of each file to the database, then loop again through each file uploading the second row of each file to the database. However, I'm stuck in having the csv files ready to be uploaded in a single object. The error happens at 'csv_data[i] = csv.reader...' Each file is for a different table, so I cannot append them.

import csv
import sys

i = 0
for argv in sys.argv[1:]:
    csv_file = open(argv, newline='', encoding='utf-8-sig')
    csv_data[i] = csv.reader(csv_file, dialect='excel', delimiter=',', quotechar='|')
    csv_file.close()
    i += 1

After this code, I would need something to loop through each file uploading a certain row number.

C.Nivs · Accepted Answer · 2021-10-25 01:51:11Z

1

zip together the files, iterate through them:

file_handles = [open(file, newline='', encoding='utf-8-sig') for file in argv[1:]]
readers =  (csv.reader(file, dialect='excel', delimiter=',', quotechar='|') for file in file_handles)

# zip here
for line_group in zip(*readers):
    # line_group is a tuple of line i of each file


# don't forget to close your files
for file_handle in file_handles:
    try:
        file_handle.close()
    except:
        print("Issue closing one of the files")

edited Oct 25, 2021 at 1:51

answered Oct 25, 2021 at 1:45

C.Nivs

13.2k3 gold badges21 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Herberts Over a year ago

Thanks a lot! This works perfectly for me!

Collectives™ on Stack Overflow

How to load multiple csv files to an iterable variable in Python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related