Latency problems

[Kyle Baley]

We've been having severe latency issues for about an hour now. Haven't made any changes to our system for some time. We have a couple of apps and it *appears* to be affecting only one at the moment. (bookedin-prod.appspot.com)

Is anyone else having trouble? We've got some pretty testy clients on our hands at the moment and I need to tell them something.

[Bartosz Kołodziej]

We have the same problems with one of our apps.

[troberti]

We had something similar yesterday morning for one of our apps. Lasted for about 3 hours. 

Today, on a different app, I saw a lot of very slow timeouts (60+) on some requests handlers which caused hundreds of extra instances to be launched.. So something is still going on.

On Thursday, June 27, 2013 5:02:49 PM UTC+2, Kyle Baley wrote:

[Kyle Baley]

Ya, the list of recent downtime notifications doesn't inspire a lot of confidence but this is the first time it's affected us dramatically in some time. In any case, we seem to be more stable now for about an hour so fingers are crossed. At the very least, I can rest assured that system reliability is a top priority at Google.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/60Q1aAlAx_I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Francois Masurel]

Yep, you're right, they should at least change their automated message template from time to time ;-)

[GregF]

Same thing here - see my post further down the list. We have two virtually identical apps for NZ and Australia, and this affected only the NZ one. Latency (even for static resurces) went up by ten times, causing lots of timeouts and warnings that processes "had a problem". It now seems to have stabilised, but latency is still higher than normal.

Instances also went through the roof, but strangely billable instances didn't. It would be interesting to know why not.

Final point - reportability. Props to the Appengine team that this is the first time I've looked for it for a couple of years, but the "Report production issue" button seems to have disappeared. Is a mute Google group the only way to communicate? Our apps are billed.

[burlug]

About latency problems and reporting issues:

Recurring latency problems just past 'billing reset'-time seems to occur/are reported since the end of february- and guess what: not any reaction from Google. Probably too busy with their funny glasses.....

https://code.google.com/p/googleappengine/issues/detail?id=9531&can=5&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log

Op donderdag 27 juni 2013 22:05:41 UTC+2 schreef GregF het volgende:

[GregF]

That URL doesn't work for me - have they deleted that issue? Any more details would be very useful.

[Aleksei Rovenski]

We are having now 20-40 instances where normally there are 2-4 for this load, so went up about 10 times. 

Just look at this pathetic screenshot http://clip2net.com/s/5iO2VD

And look how nicely billing instances grow http://clip2net.com/s/5iO47p while requests per second just a bit http://clip2net.com/s/5iO5j2

What a brilliant business model.... Service breaks, customers pay.

Any clue somebody is fixing this right now?

четверг, 27 июня 2013 г., 18:02:49 UTC+3 пользователь Kyle Baley написал:

[Pravanjan Niranjan]

I had the same problem 2 days back .

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Thanks,

Pravanjan

Dev formcreator

[Aleksei Rovenski]

I take my words back, my particular case was my own fault. It was caused by denormalized entities that we did not clean up and they gradually became so big that consumed all the memory. Instances did not die as i would have expected, instead the latency dropped and gae spinned more instances somehow limiting even those that did not run out of memory yet. Its a shame, but gae fault was the first thing that came to my mind when the problem emerged, it was not like that a year ago :(

пятница, 28 июня 2013 г., 9:14:12 UTC+3 пользователь Aleksei Rovenski написал:

[jon]

This happened to me once. Latency went up significantly and so did my bill. I challenged it but got a response along the lines of "that's just the way it is". I'm a long time GAE user and I've gone from really liking GAE to someone who constantly worries about downtime/latency and worse yet, paying more for it.

[o...@haitov.com]

I encountered the same issue yesterday,

Crazy response time of 30k-60kMS for basic requests.

The even more troubling issue is that many of the requests are incoming phone calls the comes from twillio, and when my app doesn't respond in 10s the customers hear "an application error has occurred" because that's the twillio timeout for a request.

That makes me look very very bad in front of my customers !! Moreover they aren't able to perform what they wish for which is a critical issue !!! i'm thinking seriously of migrating from app engine if this kind of downtime will continue.

I am very disapointed from the GAE team, ok there is no free support, but at least respond in the group or stack overflow about such critical issues !!!! i definitely consider this as a downtime, denying that it's like saying that a car with a flat tire is a functional car, ok you can drive with it very very slow, but it's still doesn't make it a functional car !!!!!

GAE team, please respect your customers and respond with some feedback, estimated time when this issue will be fixed, or otherwise i believe many other customer will also start looking fo other options (and there are plenty of them...)

[Kyle Baley]

Not sure if it's related but we've seen a significant spike in the number of front-end instance hours used by the system. More than one might attribute to traffic growth. It's jumped about 60-90% since the 25th. I.e. it went from 25-30 hours/day to 40-60 hours/day on average. Not gradually, either.

Is there any way to get someone to look at this to at least identify if the problem is with Google or us? That's not meant to sound snarky, I would honestly like to know.