How to skip reading empty files with pandas in Python

Question

I read all the files in one folder one by one into a pandas.DataFrame and then I check them for some conditions. There are a few thousand files, and I would love to make pandas raise an exception when a file is empty, so that my reader function would skip this file.

I have something like:

class StructureReader(FileList):
    def __init__(self, dirname, filename):
        self.dirname=dirname
        self.filename=str(self.dirname+"/"+filename)
    def read(self):
        self.data = pd.read_csv(self.filename, header=None, sep = ",")
        if len(self.data)==0:
           raise ValueError
class Run(object):
    def __init__(self, dirname):
        self.dirname=dirname
        self.file__list=FileList(dirname)
        self.result=Result()
    def run(self):
        for k in self.file__list.file_list[:]:
            self.b=StructureReader(self.dirname, k)
            try:
                self.b.read()
                self.b.find_interesting_bonds(self.result)
                self.b.find_same_direction_chain(self.result)
            except ValueError:
                pass

Regular file that I'm searching for some condition looks like:

"A/C/24","A/G/14","WW_cis",,
"B/C/24","A/G/15","WW_cis",,
"C/C/24","A/F/11","WW_cis",,
"d/C/24","A/G/12","WW_cis",,

But somehow I don't ever get ValueError raised, and my functions are searching empty files, which gives me a lot of "Empty DataFrame ..." lines in my results file. How can I skip empty files?

isn't this a dupe of this: stackoverflow.com/questions/2507808/… — EdChum
– EdChum, Commented Mar 21, 2016 at 14:38
Possible duplicate of Python - Check whether a file is empty or not — eyllanesc
– eyllanesc, Commented Apr 10, 2018 at 17:08

Ali Ben Messaoud · Accepted Answer · 2019-12-20 19:38:44Z

14

I'd first check if the file is empty, and if it isn't empty I'll try to use it with pandas. Following this link https://stackoverflow.com/a/15924160/5088142 you can find a nice way to check if a file is empty:

import os
def is_non_zero_file(fpath):  
    return os.path.isfile(fpath) and os.path.getsize(fpath) > 0

edited Dec 20, 2019 at 19:38

Ali Ben Messaoud

11.9k9 gold badges57 silver badges89 bronze badges

answered Mar 21, 2016 at 14:37

Yaron

10.6k9 gold badges50 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

JStrahl Over a year ago

With pathlib check the size with fpath = Path(fpath) and fpath.stat().st_size == 0

ivarejao · Accepted Answer · 2023-02-16 16:54:13Z

6

You can get your work done with following code, just add your CSVs path to the path variable, and run. You should get an object raw_data which is a Pandas dataframe.

import os, pandas as pd, glob
import pandas.io.common

path = "/home/username/data_folder"
files_list = glob.glob(os.path.join(path, "*.csv"))

for i in range(0,len(files_list)):
   try:
       raw_data = pd.read_csv(files_list[i])
   except pandas.errors.EmptyDataError:
      print(files_list[i], " is empty and has been skipped.")

edited Feb 16, 2023 at 16:54

ivarejao

358 bronze badges

answered Jan 7, 2018 at 7:55

Ahmad M.

52410 silver badges23 bronze badges

Comments

Community · Accepted Answer · 2017-05-23 12:01:46Z

4

You should not use pandas, but directly the python libraries. The answer is there: python how to check file empty or not

edited May 23, 2017 at 12:01

CommunityBot

11 silver badge

answered Mar 21, 2016 at 14:34

DevShark

9,18810 gold badges36 silver badges68 bronze badges

Comments

Nick Mortimer · Accepted Answer · 2019-04-21 07:19:56Z

1

How about this

files = glob.glob('*.csv')
files = list(filter(lambda file: os.stat(file).st_size > 0, files))
data = pd.read_csv(files)

answered Apr 21, 2019 at 7:19

Nick Mortimer

211 bronze badge

Collectives™ on Stack Overflow

How to skip reading empty files with pandas in Python

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related