Python | Read from multiple csv files and add to a nested list

Question

I have csv files (present in the same directory) like these:

File1:

Id,Param1,Param2
1,10,12
2,16,18
3,24,28
4,22,26

File2:

Id,Param1,Param2
1,13,19
2,15,23
3,21,25

I want to read the files and create nested lists like this:

Param1 = [[10, 16, 24, 22], [13, 15, 21]]
Param2 = [[12, 18, 28, 26], [19, 23, 25]]

What I tried:

for i in range(1,nof+1,1):
    with open("File%i.csv" %i, "rb") as f1:
        reader = csv.reader(f1)

        for row in reader:
            Param1.append(row[1])
            Param2.append(row[2])

Finally:

[Param1[i:i + n] for i in range(0, len(Param1), n)]
[Param2[i:i + n] for i in range(0, len(Param2), n)]

Would work fine if I had the same number of rows in all my files, but that isn't the case. My files have unequal number of rows. So, can someone please help me figure out how to go about creating these splits. Many thanks.

your inputs aren't csv files. please edit to show actual input, not their excel representaton — Jean-François Fabre
– Jean-François Fabre ♦, Commented Jul 23, 2018 at 16:30
I don't understand. The "actual input" would be the comma separated values of these same "excel format" numbers. — J.Calc
– J.Calc, Commented Jul 23, 2018 at 16:43
exactly. So we can reproduce & fix your issue without having to write those inputs ourselves. — Jean-François Fabre
– Jean-François Fabre ♦, Commented Jul 23, 2018 at 16:45

Wamadahama · Accepted Answer · 2018-07-23 16:39:33Z

1

Can you use pandas?

import pandas as pd

dfs = []
nof = 2
for i in range(1, nof+1, 1):
    dfs.append(pd.read_csv("File{}.csv".format(i)))

param1_list = [list(df['Param1']) for df in dfs]
param2_list = [list(df['Param2']) for df in dfs] 

print(param1_list)
print(param2_list)

try it here

answered Jul 23, 2018 at 16:39

Wamadahama

1,74814 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

J.Calc Over a year ago

I was trying to avoid using pandas, but I have to admit, it's pretty neat.

madmatrix · Accepted Answer · 2018-07-23 16:46:46Z

1

Modified the sample input a bit.

cat file1
1|10|12
2|16|18
3|24|28
4|22|26

cat file2
1|13|19
2|15|23
3|21|25

Sample code

def process(filename):
    first_list = []
    second_list = []
    with open(filename, 'r') as fh:
        for line in fh:
            line = line.rstrip()
            dummy, first_field, second_field = line.split('|')
            first_list.append(first_field)
            second_list.append(second_field)

        return [first_list, second_list]

print (process('file1'))
print (process('file2'))

Output

[['10', '16', '24', '22'], ['12', '18', '28', '26']]
[['13', '15', '21'], ['19', '23', '25']]

answered Jul 23, 2018 at 16:46

madmatrix

2651 gold badge4 silver badges15 bronze badges

Comments

jpp · Accepted Answer · 2018-07-23 16:54:50Z

Here's one way using dictionaries and csv.reader:

from io import StringIO
import csv

file1 = StringIO("""Id Param1 Param2
1   10     12
2   16     18
3   24     28
4   22     26""")

file2 = StringIO("""Id Param1 Param2
1   13     19
2   15     23
3   21     25""")

res = {}
for i, file in enumerate([file1, file2]):
    # replace file with open('...', 'r')
    with file as fin:
        reader = csv.reader(file, delimiter=' ', skipinitialspace=True)
        next(reader)  # exclude header row
        res[i] = {idx: list(map(int, x)) for idx, x in enumerate(zip(*reader))}

Param1 = [res[0][1], res[1][1]]
Param2 = [res[0][2], res[1][2]]

print(Param1, Param2, sep='\n')

[[10, 16, 24, 22], [13, 15, 21]]
[[12, 18, 28, 26], [19, 23, 25]]

G_M · Accepted Answer · 2018-07-23 17:24:48Z

1

>>> from collections import defaultdict
... from csv import DictReader
... 
... 
... def solution(filenames):
...     result = defaultdict(list)
...     for filename in filenames:
...         d = defaultdict(list)
...         with open(filename, 'r') as f:
...             reader = DictReader(f)
...             for line in reader:
...                 for k, v in line.items():
...                     d[k].append(int(v))
... 
...         for k, v in d.items():
...             result[k].append(v)
...     return result
... 
>>> result = solution(['file1.csv', 'file2.csv'])
>>> result['Param1']
[[10, 16, 24, 22], [13, 15, 21]]
>>> result['Param2']
[[12, 18, 28, 26], [19, 23, 25]]

answered Jul 23, 2018 at 17:24

G_M

3,3821 gold badge11 silver badges24 bronze badges

Collectives™ on Stack Overflow

Python | Read from multiple csv files and add to a nested list

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related