I have a relatively simple app and I've noticed significant
performance degredation over the last week. Even on extremely simple
pages like the "about" page which has the following code:
Yesterday, requests to this page were taking anywhere from 10-15
seconds to display from the time appengine first recorded the request
arrival - just to verify I adding some logging and it was almost the
entire time of the request before flow control was in my class.
Naturally once my code was executing it finished very quickly as all
it does is render a static html file. Some requests finished almost
instantaneously, but consistently I ran into the enormous delay. This
pattern leads me to believe that I'm being affected by the "cold
start" phenomenon but I would expect that as I continued to hit
refresh to generate logging data that all subsequent loads would be
fast, but many were not. Any insights?
For a simple webapp app, even cold start should be not more than
50-100ms extra. One thing you might be affected by is slow import bug;
you may want to add more logging statements to see if that is the
cause. Either way try to bring it to the attention of googlers so they
can look into it as there is probably nothing you can do.
On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> I have a relatively simple app and I've noticed significant
> performance degredation over the last week. Even on extremely simple
> pages like the "about" page which has the following code:
> Yesterday, requests to this page were taking anywhere from 10-15
> seconds to display from the time appengine first recorded the request
> arrival - just to verify I adding some logging and it was almost the
> entire time of the request before flow control was in my class.
> Naturally once my code was executing it finished very quickly as all
> it does is render a static html file. Some requests finished almost
> instantaneously, but consistently I ran into the enormous delay. This
> pattern leads me to believe that I'm being affected by the "cold
> start" phenomenon but I would expect that as I continued to hit
> refresh to generate logging data that all subsequent loads would be
> fast, but many were not. Any insights?
We have a webapp application that occasionally gets
DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
occurrence on our Django-based apps.
Happens at the usual times: say, 6-9a PST.
j
On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> For a simple webapp app, even cold start should be not more than
> 50-100ms extra. One thing you might be affected by is slow import bug;
> you may want to add more logging statements to see if that is the
> cause. Either way try to bring it to the attention of googlers so they
> can look into it as there is probably nothing you can do.
> On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > Hi -
> > I have a relatively simple app and I've noticed significant
> > performance degredation over the last week. Even on extremely simple
> > pages like the "about" page which has the following code:
> > Yesterday, requests to this page were taking anywhere from 10-15
> > seconds to display from the time appengine first recorded the request
> > arrival - just to verify I adding some logging and it was almost the
> > entire time of the request before flow control was in my class.
> > Naturally once my code was executing it finished very quickly as all
> > it does is render a static html file. Some requests finished almost
> > instantaneously, but consistently I ran into the enormous delay. This
> > pattern leads me to believe that I'm being affected by the "cold
> > start" phenomenon but I would expect that as I continued to hit
> > refresh to generate logging data that all subsequent loads would be
> > fast, but many were not. Any insights?
> We have a webapp application that occasionally gets
> DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
> occurrence on our Django-based apps.
> Happens at the usual times: say, 6-9a PST.
> j
> On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> > For a simple webapp app, even cold start should be not more than
> > 50-100ms extra. One thing you might be affected by is slow import bug;
> > you may want to add more logging statements to see if that is the
> > cause. Either way try to bring it to the attention of googlers so they
> > can look into it as there is probably nothing you can do.
> > On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > > Hi -
> > > I have a relatively simple app and I've noticed significant
> > > performance degredation over the last week. Even on extremely simple
> > > pages like the "about" page which has the following code:
> > > Yesterday, requests to this page were taking anywhere from 10-15
> > > seconds to display from the time appengine first recorded the request
> > > arrival - just to verify I adding some logging and it was almost the
> > > entire time of the request before flow control was in my class.
> > > Naturally once my code was executing it finished very quickly as all
> > > it does is render a static html file. Some requests finished almost
> > > instantaneously, but consistently I ran into the enormous delay. This
> > > pattern leads me to believe that I'm being affected by the "cold
> > > start" phenomenon but I would expect that as I continued to hit
> > > refresh to generate logging data that all subsequent loads would be
> > > fast, but many were not. Any insights?
> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?
It does
However you are not guarunteed to find anything in the cache, and I
think its indicative of a capacity problem in the early morning US
time
that startup times that normally are around 3-4 sec, blow out to 30
sec are occuring very frequently.
So the best example of this problem is one that doesn't load a big
stack and still blows out instance startup time, as per the OP's post.
<w.k.r.montgom...@googlemail.com> wrote:
> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?
> > We have a webapp application that occasionally gets
> > DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
> > occurrence on our Django-based apps.
> > Happens at the usual times: say, 6-9a PST.
> > j
> > On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> > > For a simple webapp app, even cold start should be not more than
> > > 50-100ms extra. One thing you might be affected by is slow import bug;
> > > you may want to add more logging statements to see if that is the
> > > cause. Either way try to bring it to the attention of googlers so they
> > > can look into it as there is probably nothing you can do.
> > > On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > > > Hi -
> > > > I have a relatively simple app and I've noticed significant
> > > > performance degredation over the last week. Even on extremely simple
> > > > pages like the "about" page which has the following code:
> > > > Yesterday, requests to this page were taking anywhere from 10-15
> > > > seconds to display from the time appengine first recorded the request
> > > > arrival - just to verify I adding some logging and it was almost the
> > > > entire time of the request before flow control was in my class.
> > > > Naturally once my code was executing it finished very quickly as all
> > > > it does is render a static html file. Some requests finished almost
> > > > instantaneously, but consistently I ran into the enormous delay. This
> > > > pattern leads me to believe that I'm being affected by the "cold
> > > > start" phenomenon but I would expect that as I continued to hit
> > > > refresh to generate logging data that all subsequent loads would be
> > > > fast, but many were not. Any insights?
> > Also for a static page such as about, does it not make sense to cache the
> > rendered template in Memcache, to save the processing?
> It does
> However you are not guarunteed to find anything in the cache, and I
> think its indicative of a capacity problem in the early morning US
> time
> that startup times that normally are around 3-4 sec, blow out to 30
> sec are occuring very frequently.
> So the best example of this problem is one that doesn't load a big
> stack and still blows out instance startup time, as per the OP's post.
> Rgds
> T
> On Nov 3, 5:53 am, William Montgomery
> <w.k.r.montgom...@googlemail.com> wrote:
> > Also for a static page such as about, does it not make sense to cache the
> > rendered template in Memcache, to save the processing?
> > > We have a webapp application that occasionally gets
> > > DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
> > > occurrence on our Django-based apps.
> > > Happens at the usual times: say, 6-9a PST.
> > > j
> > > On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> > > > For a simple webapp app, even cold start should be not more than
> > > > 50-100ms extra. One thing you might be affected by is slow import bug;
> > > > you may want to add more logging statements to see if that is the
> > > > cause. Either way try to bring it to the attention of googlers so they
> > > > can look into it as there is probably nothing you can do.
> > > > On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > > > > Hi -
> > > > > I have a relatively simple app and I've noticed significant
> > > > > performance degredation over the last week. Even on extremely simple
> > > > > pages like the "about" page which has the following code:
> > > > > Yesterday, requests to this page were taking anywhere from 10-15
> > > > > seconds to display from the time appengine first recorded the request
> > > > > arrival - just to verify I adding some logging and it was almost the
> > > > > entire time of the request before flow control was in my class.
> > > > > Naturally once my code was executing it finished very quickly as all
> > > > > it does is render a static html file. Some requests finished almost
> > > > > instantaneously, but consistently I ran into the enormous delay. This
> > > > > pattern leads me to believe that I'm being affected by the "cold
> > > > > start" phenomenon but I would expect that as I continued to hit
> > > > > refresh to generate logging data that all subsequent loads would be
> > > > > fast, but many were not. Any insights?
That's an excellent suggestion.
It presently isn't happening (no code change on my part, so it may be
that google fixed the prob), but if it ever does occur, I should be
able to determine whether it is an import prob.
Thanks again!
-Blake
On Nov 2, 12:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> For a simple webapp app, even cold start should be not more than
> 50-100ms extra. One thing you might be affected by is slow import bug;
> you may want to add more logging statements to see if that is the
> cause. Either way try to bring it to the attention of googlers so they
> can look into it as there is probably nothing you can do.
> On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > Hi -
> > I have a relatively simple app and I've noticed significant
> > performance degredation over the last week. Even on extremely simple
> > pages like the "about" page which has the following code:
> > Yesterday, requests to this page were taking anywhere from 10-15
> > seconds to display from the time appengine first recorded the request
> > arrival - just to verify I adding some logging and it was almost the
> > entire time of the request before flow control was in my class.
> > Naturally once my code was executing it finished very quickly as all
> > it does is render a static html file. Some requests finished almost
> > instantaneously, but consistently I ran into the enormous delay. This
> > pattern leads me to believe that I'm being affected by the "cold
> > start" phenomenon but I would expect that as I continued to hit
> > refresh to generate logging data that all subsequent loads would be
> > fast, but many were not. Any insights?
Actually, read my post again - the vast majority of the time to
process the request occurred prior to entering the code for the About
class. Using memcache to save me a few milliseconds on rendering such
a tiny page is not a priority when it's 10-15 seconds before memcache
can even do a lookup.
<w.k.r.montgom...@googlemail.com> wrote:
> Also for a static page such as about, does it not make sense to cache the
> rendered template in Memcache, to save the processing?
> > We have a webapp application that occasionally gets
> > DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
> > occurrence on our Django-based apps.
> > Happens at the usual times: say, 6-9a PST.
> > j
> > On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> > > For a simple webapp app, even cold start should be not more than
> > > 50-100ms extra. One thing you might be affected by is slow import bug;
> > > you may want to add more logging statements to see if that is the
> > > cause. Either way try to bring it to the attention of googlers so they
> > > can look into it as there is probably nothing you can do.
> > > On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > > > Hi -
> > > > I have a relatively simple app and I've noticed significant
> > > > performance degredation over the last week. Even on extremely simple
> > > > pages like the "about" page which has the following code:
> > > > Yesterday, requests to this page were taking anywhere from 10-15
> > > > seconds to display from the time appengine first recorded the request
> > > > arrival - just to verify I adding some logging and it was almost the
> > > > entire time of the request before flow control was in my class.
> > > > Naturally once my code was executing it finished very quickly as all
> > > > it does is render a static html file. Some requests finished almost
> > > > instantaneously, but consistently I ran into the enormous delay. This
> > > > pattern leads me to believe that I'm being affected by the "cold
> > > > start" phenomenon but I would expect that as I continued to hit
> > > > refresh to generate logging data that all subsequent loads would be
> > > > fast, but many were not. Any insights?
The strategy I have taken (and this works most of the time)
is in my main I import almost nothing except the bare minimum of app
engine apis
import logging
logging.getLogger().setLevel(logging.DEBUG)
import os, sys
from google.appengine.ext.webapp import Request,RequestHandler,
Response
from google.appengine.api import users
from google.appengine.api import memcache
from google.appengine.runtime import DeadlineExceededError
from google.appengine.ext.webapp.util import run_wsgi_app
Thats it,
Then in main I try and serve the page from memcache for anon users,
if the page isn't present I then start my imports for the full
application stack.
I have found that if the page in question is in memcahce I never get
the DeadLineExceededError, when they are occuring.
and I know they are occuring, If a logged user at tries at the same
time (they never get a cached page ) I am getting
DeadLineExceededErrors.
This behaviour is very consistent when we get hit with a long period
of slow instance spin ups which result in deadlines exceeded.
I am not using Django or webapp but my typical cold start time is
around 4sec. So I am experiencing a bit of grief too.
But almost never for a cached page. (Except when the system gets so
bad that even static handlers fail with DeadLineExceeded)
I am not sure how applicable this approach is for other app stacks.
Rgds
T
On Nov 4, 2:02 am, Blake <blake.commag...@gmail.com> wrote:
> Actually, read my post again - the vast majority of the time to
> process the request occurred prior to entering the code for the About
> class. Using memcache to save me a few milliseconds on rendering such
> a tiny page is not a priority when it's 10-15 seconds before memcache
> can even do a lookup.
> -Blake
> On Nov 2, 1:53 pm, William Montgomery
> <w.k.r.montgom...@googlemail.com> wrote:
> > Also for a static page such as about, does it not make sense to cache the
> > rendered template in Memcache, to save the processing?
> > > We have a webapp application that occasionally gets
> > > DeadlineExceededErrors on cold start (i.e., >30s!). This is a regular
> > > occurrence on our Django-based apps.
> > > Happens at the usual times: say, 6-9a PST.
> > > j
> > > On Nov 2, 2:17 pm, Brandon Thomson <gra...@gmail.com> wrote:
> > > > For a simple webapp app, even cold start should be not more than
> > > > 50-100ms extra. One thing you might be affected by is slow import bug;
> > > > you may want to add more logging statements to see if that is the
> > > > cause. Either way try to bring it to the attention of googlers so they
> > > > can look into it as there is probably nothing you can do.
> > > > On Nov 1, 5:48 pm, Blake <blake.commag...@gmail.com> wrote:
> > > > > Hi -
> > > > > I have a relatively simple app and I've noticed significant
> > > > > performance degredation over the last week. Even on extremely simple
> > > > > pages like the "about" page which has the following code:
> > > > > Yesterday, requests to this page were taking anywhere from 10-15
> > > > > seconds to display from the time appengine first recorded the request
> > > > > arrival - just to verify I adding some logging and it was almost the
> > > > > entire time of the request before flow control was in my class.
> > > > > Naturally once my code was executing it finished very quickly as all
> > > > > it does is render a static html file. Some requests finished almost
> > > > > instantaneously, but consistently I ran into the enormous delay. This
> > > > > pattern leads me to believe that I'm being affected by the "cold
> > > > > start" phenomenon but I would expect that as I continued to hit
> > > > > refresh to generate logging data that all subsequent loads would be
> > > > > fast, but many were not. Any insights?
|
