Using a loop to open and process txt files with csv

Question

I have data (mixed text and numbers in txt files) and I'd like to write a for loop that creates a list of lists, such that I can process the data from all the files using fewer lines.

So far I have written this:

import csv

path = (some path...)
files = [path + 'file1.txt',path + 'file2.txt', path + 
'file3.txt', ...]

for i in files:
    with open(i, 'r') as j:
        Reader = csv.reader(j)
        List = [List for List in Reader]

I think I overwrite List instead of creating a nested list, since I get Reader with size of 1 and a list that's with dimensions for one of the files.

My questions:

Given that the files may contain different line numbers, is it the right approach to save some lines of code? (What could be done better?)
I think the problem is in [List for List in Reader], is there a way to change it so I don't overwrite List? Something like add to List?

martineau · Accepted Answer · 2020-12-06 18:45:10Z

1

You can use the list append() method to add to an existing list. Since csv.reader instances are iterable objects, you can just pass one of them to the method as shown below:

import csv
from pathlib import Path

path = Path('./')
filenames = ['in_file1.txt', 'in_file2.txt']  # etc ...
List = []

for filename in filenames:
    with open(path / filename, 'r', newline='') as file:
        List.append(list(csv.reader(file)))

print(List)

Update

An even more succinct way to do it would be to use something called a "list comprehension":

import csv
from pathlib import Path

path = Path('./')
filenames = ['in_file1.txt', 'in_file2.txt']  # etc ...
List = [list(csv.reader(open(path / filename, 'r', newline='')))
            for filename in filenames]

print(List)

edited Dec 6, 2020 at 18:45

answered Dec 5, 2020 at 16:51

martineau

124k29 gold badges181 silver badges319 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Toma Over a year ago

Good idea with Path. The List created is not a list of lists as requested and would make it hard to differentiate between files.

martineau Over a year ago

Oh…in that case I guess you want a list-of-lists-of-lists. See update. Also, yes, using pathlib is often a good way to do things — I'm trying to get in the habit of using it instead of os.path.

avarice · Accepted Answer · 2020-12-05 16:19:44Z

1

Yes, use .append():

import numpy as np
import matplotlib.pyplot as plt
import csv
path = (some path...)
files = [path+x for x in ['FILESLIST']]

for i in files:
    with open(i, 'r') as j:
        Reader = csv.reader(j)
        List.append([L for L in Reader])

answered Dec 5, 2020 at 16:19

avarice

15.6k3 gold badges19 silver badges38 bronze badges

Collectives™ on Stack Overflow

Using a loop to open and process txt files with csv

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related