Fast way to Hash Numpy objects for Caching

Question

Implementing a system where, when it comes to the heavy mathematical lifting, I want to do as little as possible.

I'm aware that there are issues with memoisation with numpy objects, and as such implemented a lazy-key cache to avoid the whole "Premature optimisation" argument.

def magic(numpyarg,intarg):
    key = str(numpyarg)+str(intarg)

    try:
        ret = self._cache[key]
        return ret
    except:
        pass

    ... here be dragons ...
    self._cache[key]=value
    return value

but since string conversion takes quite a while...

t=timeit.Timer("str(a)","import numpy;a=numpy.random.rand(10,10)")
t.timeit(number=100000)/100000 = 0.00132s/call

What do people suggest as being 'the better way' to do it?

possible duplicate of How to hash a large object (dataset) in Python? — tacaswell
– tacaswell, Commented Mar 19, 2014 at 16:43
Note that str(a) only shows a part of the array, as would be later pointed out in this comment: stackoverflow.com/questions/16589791/… — paperskilltrees
– paperskilltrees, Commented Apr 11, 2023 at 21:20

Community · Accepted Answer · 2017-05-23 11:46:11Z

30

Borrowed from this answer... so really I guess this is a duplicate:

>>> import hashlib
>>> import numpy
>>> a = numpy.random.rand(10, 100)
>>> b = a.view(numpy.uint8)
>>> hashlib.sha1(b).hexdigest()
'15c61fba5c969e5ed12cee619551881be908f11b'
>>> t=timeit.Timer("hashlib.sha1(a.view(numpy.uint8)).hexdigest()", 
                   "import hashlib;import numpy;a=numpy.random.rand(10,10)") 
>>> t.timeit(number=10000)/10000
2.5790500640869139e-05

edited May 23, 2017 at 11:46

CommunityBot

11 silver badge

answered Mar 22, 2011 at 4:21

senderle

152k36 gold badges218 silver badges244 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jorgeca Over a year ago

Nice! For multidimensional arrays this gives a different hash (for the "same" array) depending on whether it's fortran or c contiguous. If that's an issue, calling np.ascontiguousarray should solve it.

Cong Ma Over a year ago

Not sure why a known slow hash function sha1 is chosen. SHA-1 is OK for minimising hash collision but poor at speed. For speed you'll need something like murmurhash or xxhash (the latter claims to be even faster).

senderle Over a year ago

@CongMa, thanks for the extra info. There are lots of options! But as you'll notice, this is already two orders of magnitude faster. And speed is never the only concern. It's probably worth using a well-understood hash if the alternative is only a few millionths of a second faster.

Erik · Accepted Answer · 2021-09-16 21:40:14Z

7

There is a package for this called joblib. Found from this question.

from joblib import Memory
location = './cachedir'
memory = Memory(location)

# Create caching version of magic
magic_cached = memory.cache(magic)
result = magic_cached(...)

# Or (for one-time use)
result = memory.eval(magic, ...)

edited Sep 16, 2021 at 21:40

Erik

2,6202 gold badges17 silver badges26 bronze badges

answered Mar 22, 2011 at 6:55

John Salvatier

3,2475 gold badges28 silver badges31 bronze badges

1 Comment

Alex Fortin Over a year ago

It would be better to have a quote from those links copied over in your answer, in case these websites go offline.

Woltan · Accepted Answer · 2014-08-05 06:15:10Z

2

For small numpy arrays also this might be suitable:

tuple(map(float, a))

if a is the numpy array.

answered Aug 5, 2014 at 6:15

Woltan

14.1k15 gold badges80 silver badges110 bronze badges

1 Comment

Maksym Ganenko Over a year ago

Oh yes, tuple is hashable in comparison with list!

Collectives™ on Stack Overflow

Fast way to Hash Numpy objects for Caching

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related