Python: Parsing multiple csv files and skip files without a keyword

Question

I am trying to read some .csv field data on python for post-processing, I typically just use something like:

for flist in glob('*.csv'):
    df = pd.read_csv(flist, delimiter = ',')

However I need to filter through the bad files which contain "Run_Terminated" somewhere in the file and skip the file entirely. I'm still new to python so I'm not familiar with all of its functionalities, any input would be appreciated. Thank you.

Jean-François Fabre · Accepted Answer · 2017-02-06 13:56:51Z

2

What you could do is first read the file fully in memory (using a io.StringIO file-like object and look for the Run_Terminated string anywhere in the file (dirty, but should be OK),

Then pass the handle to read_csv (since you can pass a handle OR a filename) so you don't have to read it again from the file.

import pandas as pd
import glob
import io

for flist in glob('*.csv'):
    with open(flist) as f:
        data = io.StringIO()
        data.write(f.read())
    if "Run_Terminated" not in data.getvalue():
        data.seek(0)  # rewind or it won't read anything
        df = pd.read_csv(data, delimiter = ',')

answered Feb 6, 2017 at 13:56

Jean-François Fabre♦

141k24 gold badges179 silver badges246 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Raku Over a year ago

Thank you! Really appreciate the explanation.

Collectives™ on Stack Overflow

Python: Parsing multiple csv files and skip files without a keyword

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related