loop over files in python?

Question

I have lots of files that I use glob.glob to sort them to plot. My files are ordered by number: lets say from file.00000 to file.99999.

I use:

filenames = sorted(glob.glob('path/file.*'))
this_next = filenames 

for i, fname1 in enumerate(this_next):
    …

Now I would like to plot every 90 files.

Puzzled about the copy operation — why do you need both filenames and this_next? The enumeration is a non-destructive operation. Are you also after an index number (so you'd get (1, 'file.00000') and then (91, 'file.00090'), etc)? — Jonathan Leffler
– Jonathan Leffler, Commented Aug 24, 2014 at 15:46
@Jonathan Leffler yes my files are eg. file1_0.00001, file_20.00001, ... — user3527941
– user3527941, Commented Aug 24, 2014 at 15:59
Hmmm; your question says 'file.00000 to file.99999'; your comment gives a different structure for the file names (file1_0.00001, file20.00001). This is confusing to those trying to help you. So, in your enumeration clause, what ends up in i and what ends up in fname1? — Jonathan Leffler
– Jonathan Leffler, Commented Aug 24, 2014 at 16:05
Sorry for the confusion. I actually should say my files are (file_0.00001, ..., file_0.99999 and file_20.0001, ..., file_20.99999). In my first script I had to subtract to files from two directories, so I was using izip. I guess I just left it from that time. It should probably no need to keep both of them. — user3527941
– user3527941, Commented Aug 24, 2014 at 16:13

enrico.bacis · Accepted Answer · 2014-08-24 16:04:20Z

1

Sure, glob and sorted return lists, so in order to get one element every 90 you can use:

from glob import glob

for file in sorted(glob('path/file.*'))[::90]:
    ...

A more memory-efficient solution, if you are sure that glob returns sorted filenames would be to use generators:

from glob import iglob
from itertools import islice

for file in islice(iglob('path/file.*'), 0, None, 90):
    ...

But if you know that the pattern is that statis, for sure the most efficient way would be to generate the file names:

for file in ('file.%5d' % i for i in xrange(0, 99999, 90)):
    ...

answered Aug 24, 2014 at 15:43

enrico.bacis

31.9k10 gold badges90 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

So, It gave me syntax error for the first option. For second one should I use islice(iglob()) instead of sorted(glob.glob())

The first one works for me, which error do you get? The second only works if you are sure that your system returns data ordered.

filenames = sorted(glob.glob('/path/*.py_0.*')) [::90]: ^ SyntaxError: invalid syntax

@user3527941 remove the :