Modify python script to run on every file in a directory

Question

so I have a python script which takes the filename as a command argument and processes that file. However, because I have 263 files which need the same processing, I was wondering whether the command argument section could be modified with a for loop to consecutively run through all the files in a folder? Cheers, Sat

EDIT:

The code for the system argument is here:

try:
    opt_list, args = getopt.getopt(sys.argv[1:], 'r:vo:A:Cp:U:eM:')

except getopt.GetoptError, msg:
    print 'prepare_receptor4.py: %s' %msg
    usage()
    sys.exit(2)

with 'r' being the name of the file needing to be processed and the others are optional arguments. I'm not sure how to modify this with a for loop.

David Z · Accepted Answer · 2010-07-13 21:52:27Z

15

As a practical matter, whatever shell you're using probably has some syntax that can be easily used for this. In Bash, for example:

for f in *; do python myscript.py $f; done

To actually do this in Python, I'd suggest structuring your program so that the main code is in a function which takes one argument, the filename.

def process(filename):
    ...code goes here...

Then you can invoke this function like so,

for f in os.listdir(folder):
    process(f)

folder could be passed as a command-line argument, or just written into the script (if it's not something you'd be reusing).

EDIT: In response to your edit, I'd suggest just giving the filenames as regular command-line arguments, without using the -r option, so that they'll wind up in args. Then you can do

for f in args:
    process(f)

or if you would rather pass the directory name as the command-line argument,

for d in args:
    for f in os.listdir(d):
        process(f)

Alternatively, I suppose you could pass multiple instances of the -r option, and then do

for opt, arg in opt_list:
    if opt == '-r':
        process(arg)

answered Jul 13, 2010 at 21:52

David Z

133k29 gold badges264 silver badges284 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

sat_s Over a year ago

Cheers, I tried to modify the code (for fun and experience) but ended up breaking it... so I used the for loop in bash which worked a treat! Sat

joel · Accepted Answer · 2010-07-13 22:13:48Z

5

When I am working on multiple files/folders, I usually use os.walk:

import os
for root, dirs, files in os.walk(dir):
   for fname in files:
       do_something(fname)

Get your directory from getopt or optparse. Also, you can build up absolute path with the os.path.abspath if you need it.

current_file = "%s%s%s" % (os.path.abspath(root), os.path.sep, fname)
do_something(current_file)

answered Jul 13, 2010 at 22:13

joel

5273 silver badges4 bronze badges

Comments

VMDX · Accepted Answer · 2010-07-13 21:58:20Z

4

os.walk() sounds like it might work here.

def traverse_and_touch(directory, touch):
  '''
  General function for traversing a local directory. Walks through
  the entire directory, and touches all files with a specified function.
  '''
  for root, dirs, files in os.walk(directory):
    for filename in files:
      touch(os.path.join(root, filename))
  return

Now, all you need to do is pass in the directory you'd like to traverse and a function and it'll perform the code on every file.

os.walk() also traverses all sub-directories.

answered Jul 13, 2010 at 21:58

VMDX

6851 gold badge7 silver badges9 bronze badges

Comments

johntellsall · Accepted Answer · 2010-07-14 04:52:53Z

I suggest your 'main' should process each file given after the options. That is, in the "args" variable. Don't pass paths in with "-r ", this limits your flexibility. If you use os.walk() etc in the program you're requiring the system to work only on trees of files, which makes it more difficult to customize and develop.

If the program works with a list of paths, it's very easy to use in different ways. For example, you can list one data file for testing. To process a directory do "myprogram dir/*.dat". To process a tree of files use backquotes:

myprogram `find . -name "*.dat"`

Lastly you can do very cheap parallel processing. Something like:

find . -name '*.dat' | xargs -P 5 myprogram

Five copies of your program are run in parallel. No locking or forks or threads or other synchronization necessary.

(Above assumes you're on a Linux/OSX type system.)

Daenyth · Accepted Answer · 2010-07-13 21:48:36Z

1

Yes, you could modify it like that. Loop through the arguments rather than indexing the first element.

answered Jul 13, 2010 at 21:48

Daenyth

37.8k15 gold badges92 silver badges130 bronze badges

Collectives™ on Stack Overflow

Modify python script to run on every file in a directory

5 Answers 5

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

1 Comment

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related