3

If I want to work with two files I can write:

with open(fname1, 'r') as f1, open(fname2, 'r') as f2:
    # do stuff with f1 and f2

But what if I have a list of paths (say, from glob.glob)? Can I do something analogous in a list comprehension? I have in mind something like:

with [open(path, 'r') for path in paths_list] as flist:
    # do stuff with this list of open file objects

As written, this doesn't work.

1
  • 1
    If you're on version 3.3+ of Python - there's contextlib.ExitStack that you can use - see docs.python.org/3/library/… - This means that only if all files are openable does the block execute - rather than looping over each file and finding somewhere through that an open error occurs... Commented Jul 16, 2013 at 19:20

3 Answers 3

8

The object of a with statement must be a context manager. So, no, you can't do this with a list, but you might be able to do it with a custom container.

See: http://docs.python.org/2/library/contextlib.html

Or, for 3.3+ there's this: http://docs.python.org/dev/library/contextlib.html#contextlib.ExitStack (Note, as per arbarnert's answer, this can be used in 2.7, using contextlib2. See his answer for the link.)

The actual solution here is probably to If you're not going to use contextlib2 put the context manager in a loop:

for path in paths_list:
    with open(path, 'r') as f:
         #whatever
         pass

Edit: Obviously, the above will open your files one at a time. There are relatively few use cases where you need to keep an undetermined number of files open at once.

Edit: To have multiple files open at once, ExitStack is the solution you are looking for.

Sign up to request clarification or add additional context in comments.

4 Comments

It appears he wants to do something with all files at once (according to his example code), so that for loop wouldn't work.
@TimPietzcker I don't really see a basis for that inference. In any case, there are very few use cases where the files can't just be slurped into memory and processed in memory.
@TimPietzcker People write a lot of things in questions. It doesn't actually follow he needs to have more than one file open at once.
Marcin, TimPietzscker: I was interested in having all open at once. But this was a point of curiosity, not related to any actual code I'm writing. I agree that for most uses, what Marcin wrote works.
6

In 3.3+, ExitStack is definitely the answer; in fact, it's the first example given in the docs:

with ExitStack() as stack:
    files = [stack.enter_context(open(path) for path in path_list]
    for f in files:
        do_something(f)

Of course if your with body is really just a loop over files, there's no reason to do this—just put a with statement for each file inside the loop. (In fact, there's a good reason not to do this—why open a possibly unbounded number of file handles at once just to use them one at a time?) But presumably your real code needs to use multiple files at the same time.


In earlier versions, you can easily just borrow ExitStack from the 3.3 source. Backporting to 3.2 is trivial; for 2.7, you need to strip out (or rewrite, if you need it) the stuff that gets fancy with exception propagation to guarantee you the right exception contexts, but that's pretty easy.

However, an even better solution is probably to install contextlib2 off PyPI, which "provides backports of features in the latest version of the standard library’s contextlib module to earlier Python versions." Then you can just use contextlib2.ExitStack instead of contextlib.ExitStack. (In fact, contextlib2 had ExitStack, under its preliminary name ContextStack, before Python 3.3 did…)


But you can also easily build a closing_all context manager, similar to the stdlib's closing but for multiple things:

@contextlib.contextmanager
def closing_all(things):
    try:
        yield things
    finally:
        for thing in things:
            thing.close()

If you need to deal with things whose close method can raise, you need to be a little smarter—but with file objects, and most other types you'd using with closing, you don't need that.

The bigger problem is that if any open can raise an exception, it's hard to find any valid sequence you can actually pass as the things argument. But if that's not a problem, using it is even simpler than ExitStack:

with closing_all(open(path) for path in path_list) as files:
    for f in fs:
        do_something(f)

You could also build an opening_all(paths, mode='r') that does the opens for you and wraps them in a closing_all, but I don't think that adds much.

Of course if you need to do this often, the best answer is to build opening_all around ExitStack and not even bother with closing_all.

4 Comments

Does ExitStack depend on any py 3 features? Or could it be backported?
@Marcin: I don't actually know if the version checked in does… but the version from the thread and bug report definitely works on 2.7. Let me test what's there and get back to you.
@Marcin: OK, never mind, even better answer: Nick Coghlan (who wrote the version on the thread in the first place, and the earlier implementation it was based on) is maintaining a backport in contextlib2. See my edit. So, that's the right way to do it, whether you're on 3.2 or 2.x.
@APE: Well, it's new to 3.3. And, although the idea goes back to the original with proposals (see PEP 343 and its linked predecessors), it took 8 years until anyone but Nick Coghlan understood how cool it was and agreed to put it in the stdlib. So, it's not surprising that most people don't know it yet.
2

You can use fileinput:

import fileinput

for line in fileinput.input(fileList):
    ...

1 Comment

If you want to treat them all as one long file, fileinput is definitely the answer. But if you want to, e.g., un-gzip each one or something, it's not. Also you should still use a with statement if you're in 3.2+ (with fileinput.input(fileList) as f:), and if not, you want to close explicitly. Otherwise, you're leaking the last file handle, just as you would be if you called open on it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.