Increase retrieving perf python appengine with memcache

Question

Hello I want to know if in the appengine env the things I did is usefull Because I don't know why this page is very slow.

class Foo(db.Model):
    id = db.StringProperty(multiline=True)
    name = db.StringProperty(multiline=True)
    date = db.DateTimeProperty(auto_now_add=True)
    description = db.TextProperty()    
    carac = db.StringProperty(multiline=True)



class FoosPage(webapp.RequestHandler):
  def get(self):
    foos =  memcache.get("all-foos")
    if foos is None:
        foos = Foo.all()
        size = foo.count()
    else :
        size = len(foo)

    foosTab=[]
    for i in range(size) :
        foosTab.insert(i,foos[i])
        memcache.set("all-foos", foosTab, 60)
    template_values = {
        'foos': foos
            }
    path = os.path.join(os.path.dirname(__file__) + '/../templates/', 'foos.html')
    self.response.out.write(template.render(path, template_values))

Slow relative to what? Can you provide some timing data for this vs. something similar? — Joshua Hoblitt
– Joshua Hoblitt, Commented Sep 16, 2011 at 17:24
how many objects are in foo? are you running this on your dev machine or is it deployed? Does it go faster if you do a fetch(1000) right before you do the call to foo.count()? — Peter Recore
– Peter Recore, Commented Sep 16, 2011 at 17:26
Hi very slow, for about 50 object 4sec,something strange In local it is faster, @peter I will test for the fetch — Christophe Debove
– Christophe Debove, Commented Sep 16, 2011 at 17:35
Have you used appstats? It's designed to answer exactly this sort of question. — Nick Johnson
– Nick Johnson, Commented Sep 19, 2011 at 4:40

Dave W. Smith · Accepted Answer · 2011-09-17 03:48:18Z

5

You have the memcache.set() inside the loop. That's a lot of unnecessary traffic to the memcache service. Do it once, after the loop.

Further, the way you have that coded, there's no need for establishing size.

foosTab = []
for foo in foos:
    foosTab.append(foo)

or, more idiomatically

foosTab = [foo for foo in foos]

That'll save you doing a separate count().

edited Sep 17, 2011 at 3:48

answered Sep 16, 2011 at 18:19

Dave W. Smith

25k4 gold badges42 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

santiagobasulto Over a year ago

+1, good answer. Also, i usually store the entity keys instead of the hole elements.

Nick Johnson Over a year ago

Iterating over query results is a bad idea - it results in one RPC call per 20 results.

Nick Johnson · Accepted Answer · 2011-09-19 04:44:17Z

This block of code:

foos =  memcache.get("all-foos")
if foos is None:
    foos = Foo.all()
    size = foo.count()
else :
    size = len(foo)

foosTab=[]
for i in range(size) :
    foosTab.insert(i,foos[i])
    memcache.set("all-foos", foosTab, 60)

Can be replaced with this (much simpler) code:

foos = memcache.get("all-foos")
if not foos:
  foos = Foo.all.fetch(1000) # At most 1000 foos
  memcache.set("all-foos", foos, 60)

Specifically, it avoids the unnecessary call to count() (which issues an expensive RPC for something you'll find out anyway when you fetch the result), it does a single fetch call instead of iterating over the results and fetching in batches of 20, and it only calls memcache set (once!) if it had to fetch the list in the first place.

Collectives™ on Stack Overflow

Increase retrieving perf python appengine with memcache

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related