GAE cluster under high load?

[c h]

Hello,

I just started getting lots of these errors this afternoon on one of my apps:

"Request was aborted after waiting too long to attempt to service your request. This may happen sporadically when the App Engine serving cluster is under unexpectedly high or uneven load. If you see this message frequently, please contact the App Engine team."

i have not seen that before, and have not changed my app code in over a week. any ideas what might be going on? did something change with the 1.6.2 upgrade that i need to account for in my code?

thanks,

christian

[Robert Kluin]

Hi Christian,
Are/were you seeing small bursts of those or seeing a solid stream
for a prolonged period? Are you still seeing them? Did you notice
anything strange about the number of instances?

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/oUr3fUxx9K4J.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[c h]

Robert,

What i observed when it was brought to my attention was that the site had problems loading some pages.  it seemed to be isolated to a subset of pages on the site - some pages loaded well, some did not load.  a little bit after i posted the problem seems to have subsided, but so has traffic on the site.  the number of instances and latency seemed normal to me.

the site is the website and online ordering system for a restaurant in NYC.  the problem was brought to my attention when the stores reported that they had not received any online orders during first hour of the dinner rush today, and that concerned them.  that is when i looked into the problem and found the error messages in the online ordering part of the site.

does that info help?

thanks!

christian

[Brandon Wirtz]

HR or MS?

 

What is max pending latency.

 

Brandon Wirtz BlackWaterOps: President / Lead Mercenary
Work: 510-992-6548 Toll Free: 866-400-4536 IM: dra...@gmail.com (Google Talk) Skype: drakegreene YouTube: BlackWaterOpsDotCom BlackWater Ops Cloud On A String Mastermind Group

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/UKqIJVkiEEoJ.

[c h]

High replication,  max pending latency is "automatic"

[Brandon Wirtz]

Never set Max Pending latency to automatic.  3 seconds TOPs, chances are your users will thank you if you set it to 1 second.

 

Brandon Wirtz BlackWaterOps: President / Lead Mercenary
Work: 510-992-6548 Toll Free: 866-400-4536 IM: dra...@gmail.com (Google Talk) Skype: drakegreene YouTube: BlackWaterOpsDotCom BlackWater Ops Cloud On A String Mastermind Group

 

 

From: google-a...@googlegroups.com [mailto:google-a...@googlegroups.com] On Behalf Of c h
Sent: Thursday, February 02, 2012 12:19 AM
To: google-a...@googlegroups.com
Subject: Re: [google-appengine] GAE cluster under high load?

 

High replication,  max pending latency is "automatic"

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/el3SJNO6qP8J.

[c h]

thanks for the tip!  i'll try playing with that.  do you think that is at all related to the problem i was having?

[Robert Kluin]

Hey Christian,
It could just be that you hit some period of latency for a service.
Was your average latency higher than normal?

It is possible that reducing the max latency could help. That will
encourage App Engine to spin up new instances more willingly. For
apps where user experience is important I like to run with a low max
latency. Keep in mind that if your app has crappy startup times it
could be worse for the user, since they'll have to sit there waiting
on an instance.

You could also attempt to adjust your idle instances values. Think
of your idle instances as a "buffer" to handle bursts of new traffic.
There's probably no reason to have a lot of idle instances for most
apps. I've found having 5 or 10 is sufficient for apps with good
startup times and smooth (even if steep) traffic ramp ups.

Robert

On Thu, Feb 2, 2012 at 12:19, c h <how...@umich.edu> wrote:
> thanks for the tip!  i'll try playing with that.  do you think that is at
> all related to the problem i was having?
>

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/ABlOIUxQSOEJ.

[c h]

Thanks for the advice Robert!

i need to verify my instance startup time so that i can pick a good pending latency.  i have been playing with idle instances on non-production apps and have found that settings similar to your experiences are helpful.  i'm going to start rolling those out.

i didn't notice if latency had gone high during that period.  it seems to have subsided since then so i must have hit a high-load period.  it would be nifty if there was a way to detect that and give the user a better page then the GAE 500 error page.

thanks again,

cfh

[Robert Kluin]

Sounds like you're on the right track to me. And I totally agree,
having users bump into generic 500 errors sucks. You can serve a
custom error in some cases:
http://code.google.com/appengine/docs/python/config/appconfig.html#Custom_Error_Responses

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/evOukBZe5_YJ.