Python file cache

Question

I'm creating some objects from files (validators from templates xsd files, to draw together other xsd files, as it happens), and I'd like to recreate the objects when the file on disk changes.

I could create something like:

def getobj(fname, cache = {}):
    try:
        obj, lastloaded = cache[fname]
        if lastloaded < last_time_written(fname):
           # same stuff as in except clause
    except KeyError:
        obj = create_from_file(fname)
        cache[fname] = (obj, currenttime)

    return obj

However, I would prefer to use someone else's tested code if it exists. Is there an existing library that does something like this?

Update: I'm using python 2.7.1.

Note that instead of repeating the code in the except clause inside your if statement, you could just raise KeyError() instead. — Amber
– Amber, Commented Mar 24, 2012 at 19:19
@Amber Or use an inner function, which would probably be cleaner. — Marcin
– Marcin, Commented Mar 24, 2012 at 19:41
Contrary to @Katriel I don't like the mutable default argument here because I don't think they behave intuitively. Most of the time mutable default arguments change when you don't expect them to. In this case it is of course intended, but someone else reading the code may find themselves either (a) not understanding how the function works because it is counter-intuitive that cache will be anything else than {}, or (b), suspect that the function will fail at some point because it uses mutable default arguments. — sigvaldm
– sigvaldm, Commented Jun 11, 2018 at 12:36

Raymond Hettinger · Accepted Answer · 2012-03-24 19:45:43Z

5

Your code (including the cache logic) looks fine.

Consider moving the cache variable outside the function definition. That will make it possible to add other functions to clear or inspect the cache.

If you want to look at code that does something similar, look at the source for the filecmp module: http://hg.python.org/cpython/file/2.7/Lib/filecmp.py The interesting part is how the stat module is used to determine whether a file has changed. Here is the signature function:

def _sig(st):
    return (stat.S_IFMT(st.st_mode),
            st.st_size,
            st.st_mtime)

edited Mar 24, 2012 at 19:45

answered Mar 24, 2012 at 19:14

Raymond Hettinger

229k67 gold badges405 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Katriel · Accepted Answer · 2012-03-24 19:45:34Z

Three thoughts.

Use try... except... else for a neater control flow.
File modification times are notoriously unstable -- in particular, they don't necessarily correspond to the most recent time the file was modified!

Python 3 contains a caching decorator: functools.lru_cache. Here's the source.

def lru_cache(maxsize=100):
    """Least-recently-used cache decorator.

    If *maxsize* is set to None, the LRU features are disabled and the cache
    can grow without bound.

    Arguments to the cached function must be hashable.

    View the cache statistics named tuple (hits, misses, maxsize, currsize) with
    f.cache_info().  Clear the cache and statistics with f.cache_clear().
    Access the underlying function with f.__wrapped__.

    See:  http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used

    """
    # Users should only access the lru_cache through its public API:
    #       cache_info, cache_clear, and f.__wrapped__
    # The internals of the lru_cache are encapsulated for thread safety and
    # to allow the implementation to change (including a possible C version).

    def decorating_function(user_function,
                tuple=tuple, sorted=sorted, len=len, KeyError=KeyError):

        hits = misses = 0
        kwd_mark = (object(),)          # separates positional and keyword args
        lock = Lock()                   # needed because ordereddicts aren't threadsafe

        if maxsize is None:
            cache = dict()              # simple cache without ordering or size limit

            @wraps(user_function)
            def wrapper(*args, **kwds):
                nonlocal hits, misses
                key = args
                if kwds:
                    key += kwd_mark + tuple(sorted(kwds.items()))
                try:
                    result = cache[key]
                    hits += 1
                except KeyError:
                    result = user_function(*args, **kwds)
                    cache[key] = result
                    misses += 1
                return result
        else:
            cache = OrderedDict()       # ordered least recent to most recent
            cache_popitem = cache.popitem
            cache_renew = cache.move_to_end

            @wraps(user_function)
            def wrapper(*args, **kwds):
                nonlocal hits, misses
                key = args
                if kwds:
                    key += kwd_mark + tuple(sorted(kwds.items()))
                try:
                    with lock:
                        result = cache[key]
                        cache_renew(key)        # record recent use of this key
                        hits += 1
                except KeyError:
                    result = user_function(*args, **kwds)
                    with lock:
                        cache[key] = result     # record recent use of this key
                        misses += 1
                        if len(cache) > maxsize:
                            cache_popitem(0)    # purge least recently used cache entry
                return result

        def cache_info():
            """Report cache statistics"""
            with lock:
                return _CacheInfo(hits, misses, maxsize, len(cache))

        def cache_clear():
            """Clear the cache and cache statistics"""
            nonlocal hits, misses
            with lock:
                cache.clear()
                hits = misses = 0

        wrapper.cache_info = cache_info
        wrapper.cache_clear = cache_clear
        return wrapper

    return decorating_function

I never knew about the else clause. Thanks for that (and all of this).

Xavier Combelle · Accepted Answer · 2012-03-24 19:18:15Z

1

Unless there is a specific reason to use it as argument I would use cache as a global object

answered Mar 24, 2012 at 19:18

Xavier Combelle

11.3k5 gold badges30 silver badges53 bronze badges

4 Comments

Marcin Over a year ago

Valid, it was more an act of whimsy while composing in the SO window.

SingleNegationElimination Over a year ago

well, one reason is for performance. the whole purpose of a cache is to improve performance, and local variable lookups (including default arguments) are faster by a modest amount compared with global lookups. That said, this pattern is a great way to trip up future generations that aren't already familiar with this language quirk, and as you say, a global should be preferred for its explicitness when performance isn't of utmost importance.

Katriel Over a year ago

@TokenMacGuy the usual idiom is def foo(cache=cache): to copy global variables into the local scope.

Xavier Combelle Over a year ago

@TokenMacGuy I would say that the performance of global variable is pretty good comparing at a file seek

Collectives™ on Stack Overflow

Python file cache

3 Answers 3

Comments

1 Comment

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related