Open multiple files and store the data in arrays in python

Question

import numpy as np
import matplotlib.pyplot as plt
import csv

file = 'random_file'
with open(file + '.txt') as datafile:
    for line in range(7):
        next(datafile)
    for line in datafile:
        lines = datafile.readlines()
        x0_1  = [float(line.split()[0]) for line in lines]
        y1_1  = [float(line.split()[1]) for line in lines]
        Z_1   = [float(line.split()[2]) for line in lines]
        u_1   = [float(line.split()[3]) for line in lines]
        v_1   = [float(line.split()[4]) for line in lines]
        V0_1  = [float(line.split()[5]) for line in lines]
        V1_1  = [float(line.split()[6]) for line in lines]
        Yf_1  = [float(line.split()[7]) for line in lines]
        Yo_1  = [float(line.split()[8]) for line in lines]
        T_1   = [float(line.split()[9]) for line in lines]
        Yp0_1 = [float(line.split()[10]) for line in lines]
        Yp1_1 = [float(line.split()[11]) for line in lines]
        D_1   = [float(line.split()[12]) for line in lines]
        rho_1 = [float(line.split()[13]) for line in lines]
#Do something with the data

Currently, I have this code to open a single file, skip the first seven lines in the file and then store the data in the file to arrays. But I have multiple files that I would like to read and store the data in the file to arrays in a similar fashion. As of now, I am just repeating the above code for different files. How can I read all the files at once and store the data in arrays?

The data file looks something like this,

#First seven lines ignore #

  0.00000E+00  5.00000E-04  9.28613E-01  6.58135E-02  4.05154E-01  6.58135E-02  4.05154E-01  9.24508E-01  2.01228E-16  4.44076E+02  1.12880E-02  9.23567E-03  2.46667E-05  4.53488E-01
  0.00000E+00  1.50000E-03  8.14544E-01  9.64630E-02  3.82685E-01  9.64630E-02  3.82685E-01  8.03881E-01  0.00000E+00  6.74294E+02  2.93252E-02  2.39933E-02  2.46667E-05  3.15181E-01
  0.00000E+00  2.50000E-03  5.46555E-01  1.40368E-01  1.23704E-01  1.40368E-01  1.23704E-01  5.20482E-01  0.00000E+00  1.21516E+03  7.17009E-02  5.86644E-02  2.46667E-05  2.01024E-01
  0.00000E+00  3.50000E-03  1.23207E-01  1.08260E-01 -2.56808E-01  1.08260E-01 -2.56808E-01  7.27916E-02  0.00000E+00  2.06958E+03  1.38643E-01  1.13435E-01  2.46667E-05  1.54495E-01

Put this code in a function, and call it with each file you want to read. Also, instead of writing [float(line.split()[0]) for line in lines] over and over again, you can use the zip function to "transpose" the table. — koorkevani
– koorkevani, Commented May 21, 2021 at 8:18

Vosem · Accepted Answer · 2021-05-21 08:27:03Z

1

You can create an array with all of the filenames and iterate through those. As long as the files have the data stored the same way and you want to read the same thing in them.

Something like this:

files = ['file1', 'file2', 'file3']
    
for file in files:
  with open(file + '.txt') as datafile:
  # Skip the first seven lines, Read the rest of the file

answered May 21, 2021 at 8:27

Vosem

1061 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Serge Ballesta · Accepted Answer · 2021-05-21 09:01:18Z

Should be a comment since it is not a direct answer, but formatting code in comment is not possible.

Beware: your code actually skips 8 lines and is terribly inefficient. The problem is here:

for line in range(7):
    next(datafile)               # ok, 7 lines skipped
for line in datafile:            # read 8th line 
    lines = datafile.readlines() # lines only starts at the 9th line

Additionally you repeat the splitting of the line for each field which is not efficient.

As you already import the csv module, you could do:

for line in range(7):
    next(datafile)               # ok, 7 lines skipped
x0_1, y1_1, Z_1, u_1, v_1, V0_1, V1_1, Yf_1, Yo_1, T_1, Yp0_1, Yp1_1, D_1, rho_1 = [
    [float(i) for i in row] for row in zip(*csv.reader(datafile, delimiter=' ',
                                                       skipinitialspace=True))][:14]

Collectives™ on Stack Overflow

Open multiple files and store the data in arrays in python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related