struggling with memcache on google app engine python

Question

I've been struggling to get memcache working on my app for a bit now. I thought I had finally got it working where it never reads from the database (unless memcache data is lost of course), only to have my site shut down because of a over-qota number of datastore reads! I'm currently using a free appspot and would like to keep it that way for as long as possible. Anyway, here's my code, maybe somebody can help me find the hole in it.

I am currently trying to implement memcache by overriding the db.Model.all(), delete(), and put() methods to query memcache first. I have memcache set up where each object in the datastore has it's own memcache value with it's id as the key. Then for each Model class I have a list of the id's under a key it knows how to query. I hope I explained this clear enough.

""" models.py """
         @classmethod
         def all(cls, order="sent"):
                 result = get_all("messages", Message)
                 if not result or memcache.get("updatemessages"):
                         result = list(super(Message, cls).all())
                         set_all("messages", result)
                         memcache.set("updatemessages", False)
                         logging.info("DB Query for messages")

                 result.sort(key=lambda x: getattr(x, order), reverse=True)
                 return result

         @classmethod
         def delete(cls, message):
                 del_from("messages", message)
                 super(Message, cls).delete(message)

         def put(self):
                 super(Message, self).put()

                 add_to_all("messages", self)



""" helpers.py """

 def get_all(type, Class):
         all = []
         ids = memcache.get(type+"allid")
         query_amount = 0
         if ids:
                 for id in ids:
                         ob = memcache.get(str(id))
                         if ob is None:
                                 ob = Class.get_by_id(int(id))
                                 if ob is None:
                                         continue
                                 memcache.set(str(id), ob)
                                 query_amount += 1
                         all.append(ob)
                 if query_amount: logging.info(str(query_amount) + " ob queries")
                 return all
         return None

 def add_to_all(type, object):
         memcache.set(str(object.key().id()), object)
         all = memcache.get(type+"allid")
         if not all:
                 all = [str(ob.key().id()) for ob in object.__class__.all()]
                 logging.info("DB query for %s" % type)
         assert all is not None, "query returned None.  Send this error code to ____: 2 3-193A"
         if not str(object.key().id()) in all:
                 all.append(str(object.key().id()))
         memcache.set(type+"allid", all)

 @log_on_fail
 def set_all(type, objects):
         assert type in ["users", "messages", "items"], "set_all was not passed a valid type.  Send this error code to ____: 33-205"
         assert not objects is None, "set_all was passed None as the list of objects.    Send this error code to _________: 33-206"
         all = []
         for ob in objects:
                 error = not memcache.set(str(ob.key().id()), ob)
                 if error:
                         logging.warning("keys not setting properly. Object must not be pickleable")
                 all.append(str(ob.key().id()))
         memcache.set(type+"allid", all)

 @log_on_fail
 def del_from(type, object):
         all = memcache.get(type+"allid")
         if not all:
                 all = object.__class__.all()
                 logging.info("DB query %s" % type)
         assert all, "Could not find any objects.  Send this error code to _____: 13-    219"
         assert str(object.key().id()) in all, "item not found in cache.  Send this error code to ________: 33-220"
         del all[ all.index(str(object.key().id())) ]
         memcache.set(type+"allid", all)
         memcache.delete(str(object.key().id()))

I apologize for all of the clutter and lack of elegance. Hopefully somebody will be able to help. I've thought about switching to ndb but for now I rather stick to my custom cache. You'll notice the logging.info("some-number of ob queries"). I get this log quite often. Maybe once or twice every half hour. Does memcache really lose data that often or is something wrong with my code?

Is it possible to show the code without line numbers?

David Robinson
– David Robinson

2012-11-01 03:13:07 +00:00
Commented Nov 1, 2012 at 3:13 — David Robinson
– David Robinson, Commented Nov 1, 2012 at 3:13

bossylobster · Accepted Answer · 2012-11-01 05:19:18Z

5

Simple solution: switch to NDB.

NDB models will store values in memcache and in an instance cache (which is 100% free) and these models will also invalidate the cache for you when you update/delete your objects. Retrieval will first try to get from the instance cache, then if that fails from memcache, and finally from the datastore, and it will set the value in any of the caches missed on the way up.

answered Nov 1, 2012 at 5:19

bossylobster

10.2k1 gold badge44 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Harrison Over a year ago

What if I want my cache to update and delete from itself instead of invalidating it and reading the DB?

bossylobster Over a year ago

Do you mean a memcache/instance cache only implementation? The phrase update from itself doesn't make sense otherwise.

Harrison Over a year ago

Sorry, here's what I mean't to say: Does NDB "refill" or update it's entity cache from the datastore every time I delete() or put()? Is this what you mean by the models invalidating "the cache for you when you update/delete your objects"? If so, it doesn't seem like it will help too much seeing that someone "put()s" an entity quite often and it would have to reload some 1.5k instances from the database each time.

bossylobster Over a year ago

It doesn't reload, it sets the caches in the same process that it puts the data to the datastore.

Thanos Makris · Accepted Answer · 2012-11-01 08:11:54Z

0

App Engine memcache removes objects by an optimized eviction algorithm, thus having this log message with the frequency you described results in two possible explanations.

Either these data are not accessed very often or the amount of data you have in your memcache is pretty large, and thus some of it is removed from time to time.

I would also propose to move to ndb which handles the use of memcache and instance cache quite efficiently.

Hope this helps!

answered Nov 1, 2012 at 8:11

Thanos Makris

3,1152 gold badges19 silver badges26 bronze badges

Collectives™ on Stack Overflow

struggling with memcache on google app engine python

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related