Is there a way to load data from all files in a directory using Python?

Question

My question: Is there a way to load data from all files in a directory using Python

Input: Get all files in a given directory of mine (wow.txt, testting.txt,etc.)

Process: I want to run all the files through a def function

Output: I want the output to be all the files names and their respective content below it.For example:

/home/file/wow.txt "all of its content" /home/file/www.txt "all of its content"

Here is my code:

# Import Functions
import os
import sys

# Define the file path
path="/home/my_files"
file_name="wow.txt"

#Load Data Function
def load_data(path,file_name):
    """
    Input  : path and file_name
    Purpose: loading text file
    Output : list of paragraphs/documents and
             title(initial 100 words considered as title of document)
    """
    documents_list = []
    titles=[]
    with open( os.path.join(path, file_name) ,"rt", encoding='latin-1') as fin:
        for line in fin.readlines():
            text = line.strip()
            documents_list.append(text)
    print("Total Number of Documents:",len(documents_list))
    titles.append( text[0:min(len(text),100)] )
    return documents_list,titles

#Output
load_data(path,file_name)

Here is my output:

My Problem is that my output only takes one file and shows its content. Obviously, i defined the path and file name in my code to one file but I am confused as to how to write the path in a way to load all the files and output each of its contents separately. Any suggestions?

look for os.listdir() or glob, read all the files and their contents afterwards — DirtyBit
– DirtyBit, Commented Apr 17, 2019 at 15:13

DirtyBit · Accepted Answer · 2019-04-17 15:21:07Z

4

Using glob:

import glob
files = glob.glob("*.txt")           # get all the .txt files

for file in files:                   # iterate over the list of files
    with open(file, "r") as fin:     # open the file
        # rest of the code

Using os.listdir():

import os
arr = os.listdir()    
files = [x for x in arr if x.endswith('.txt')]

for file in files:                   # iterate over the list of files
    with open(file, "r") as fin:     # open the file
       # rest of the code

edited Apr 17, 2019 at 15:21

answered Apr 17, 2019 at 15:16

DirtyBit

16.8k5 gold badges37 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Julia · Accepted Answer · 2019-04-17 15:15:41Z

3

Try this:

import glob

for file in glob.glob("test/*.xyz"):
    print(file)

if my directory name was "test" and I had lots of xyz files in them...

edited Apr 17, 2019 at 15:15

answered Apr 17, 2019 at 15:14

Julia

1,1461 gold badge11 silver badges20 bronze badges

1 Comment

Julia Over a year ago

I accidentally published before finishing it :D

VnC · Accepted Answer · 2019-04-17 15:16:29Z

0

You can use glob and pandas

import pandas as pd import glob

path = r'some_directory' # use your path
all_files = glob.glob(path + "/*.txt")

li = []

for filename in all_files:
    #read file here
    # if you decide to use pandas you might need to use the 'sep' paramaeter as well
    df = pd.read_csv(filename, index_col=None, header=0)
    li.append(df)

# get it all together
frame = pd.concat(li, axis=0, ignore_index=True)

answered Apr 17, 2019 at 15:16

VnC

2,0361 gold badge19 silver badges28 bronze badges

Comments

lmiguelvargasf · Accepted Answer · 2019-04-17 15:17:12Z

0

I will take advantage of the function you have already written, so use the following:

data = []
path="/home/my_files"
dirs = os.listdir( path )
for file in dirs:
    data.append(load_data(path, file))

In this case you will have all data in the list data.

answered Apr 17, 2019 at 15:17

lmiguelvargasf

71.2k55 gold badges231 silver badges239 bronze badges

Comments

AdForte · Accepted Answer · 2019-04-17 15:21:12Z

0

Hi you can use a for loop on a listdir:

os.listdir(<path of your directory>)

this gives you the list of files in your directory, but this gives you also the name of folders in that directory

answered Apr 17, 2019 at 15:21

AdForte

3032 silver badges12 bronze badges

Comments

peterfranciscook · Accepted Answer · 2019-04-17 15:29:08Z

Try generating a file list first, then passing that to a modified version of your function.

def dir_recursive(dirName):
    import os
    import re

    fileList = list()
    for (dir, _, files) in os.walk(dirName):
         for f in files:
             path = os.path.join(dir, f)
             if os.path.exists(path):
                 fileList.append(path)

    fList = list()
    prog = re.compile('.txt$')
    for k in range(len(fileList)):
        binMatch = prog.search(fileList[k])
        if binMatch:
            fList.append(binMatch.string)

    return fList

def load_data2(file_list):
    documents_list = []
    titles=[]
    for file_path in file_list:
        with open( file_path ,"rt", encoding='latin-1') as fin:
            for line in fin.readlines():
                text = line.strip()
                documents_list.append(text)
    print("Total Number of Documents:",len(documents_list))
    titles.append( text[0:min(len(text),100)] )
    return documents_list,titles

# Generate a file list & load the data from it
file_list = dir_recursive(path)
documents_list, titles = load_data2(file_list)

Collectives™ on Stack Overflow

Is there a way to load data from all files in a directory using Python?

6 Answers 6

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related