Significant performance degredation over the last week

[Blake]

Hi -

I have a relatively simple app and I've noticed significant
performance degredation over the last week. Even on extremely simple
pages like the "about" page which has the following code:

class About(webapp.RequestHandler):

def get(self):

template_values = {}
path = os.path.join(os.path.dirname(__file__), '../views/
about.html')
self.response.out.write(template.render(path,
template_values))

Yesterday, requests to this page were taking anywhere from 10-15
seconds to display from the time appengine first recorded the request
arrival - just to verify I adding some logging and it was almost the
entire time of the request before flow control was in my class.
Naturally once my code was executing it finished very quickly as all
it does is render a static html file. Some requests finished almost
instantaneously, but consistently I ran into the enormous delay. This
pattern leads me to believe that I'm being affected by the "cold
start" phenomenon but I would expect that as I continued to hit
refresh to generate logging data that all subsequent loads would be
fast, but many were not. Any insights?

-Blake

[Brandon Thomson]

For a simple webapp app, even cold start should be not more than
50-100ms extra. One thing you might be affected by is slow import bug;
you may want to add more logging statements to see if that is the
cause. Either way try to bring it to the attention of googlers so they
can look into it as there is probably nothing you can do.

[Jason C]

We have a webapp application that occasionally gets
DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
occurrence on our Django-based apps.

Happens at the usual times: say, 6-9a PST.

j

[William Montgomery]

Also for a static page such as about, does it not make sense to cache the rendered template in Memcache, to save the processing? 

Will.

2009/11/2 Jason C <jason.a...@gmail.com>

[Tim Hoffman]

> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?

It does

However you are not guarunteed to find anything in the cache, and I
think its indicative of a capacity problem in the early morning US
time
that startup times that normally are around 3-4 sec, blow out to 30
sec are occuring very frequently.

So the best example of this problem is one that doesn't load a big
stack and still blows out instance startup time, as per the OP's post.

Rgds

T

On Nov 3, 5:53 am, William Montgomery

<w.k.r.montgom...@googlemail.com> wrote:
> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?
>
> Will.
>

> 2009/11/2 Jason C <jason.a.coll...@gmail.com>

[Brandon Thomson]

Yes, we are now seeing the occasional deadline exceeded in the AM US
time for a small-stack app as well.

[Blake]

Brandon -

That's an excellent suggestion.
It presently isn't happening (no code change on my part, so it may be
that google fixed the prob), but if it ever does occur, I should be
able to determine whether it is an import prob.
Thanks again!

-Blake

[Blake]

Actually, read my post again - the vast majority of the time to
process the request occurred prior to entering the code for the About
class. Using memcache to save me a few milliseconds on rendering such
a tiny page is not a priority when it's 10-15 seconds before memcache
can even do a lookup.

-Blake

On Nov 2, 1:53 pm, William Montgomery

<w.k.r.montgom...@googlemail.com> wrote:
> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?
>
> Will.
>

> 2009/11/2 Jason C <jason.a.coll...@gmail.com>

[Tim Hoffman]

HI Blake

The strategy I have taken (and this works most of the time)
is in my main I import almost nothing except the bare minimum of app
engine apis

import logging
logging.getLogger().setLevel(logging.DEBUG)
import os, sys
from google.appengine.ext.webapp import Request,RequestHandler,
Response
from google.appengine.api import users
from google.appengine.api import memcache
from google.appengine.runtime import DeadlineExceededError

from google.appengine.ext.webapp.util import run_wsgi_app

Thats it,

Then in main I try and serve the page from memcache for anon users,
if the page isn't present I then start my imports for the full
application stack.

I have found that if the page in question is in memcahce I never get
the DeadLineExceededError, when they are occuring.
and I know they are occuring, If a logged user at tries at the same
time (they never get a cached page ) I am getting
DeadLineExceededErrors.

This behaviour is very consistent when we get hit with a long period
of slow instance spin ups which result in deadlines exceeded.

I am not using Django or webapp but my typical cold start time is
around 4sec. So I am experiencing a bit of grief too.
But almost never for a cached page. (Except when the system gets so
bad that even static handlers fail with DeadLineExceeded)

I am not sure how applicable this approach is for other app stacks.

Rgds

T