High latency

[Francois Masurel]

Hi Google,

It feels like it's time to buy some new servers, we are getting higher and higher latency lately :

Hopefully Xmas is getting closer :-)

Francois

[Francois Masurel]

Forgot the mandatory URL : http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency

[Mos]

Yes there is a problem currently.
The last hours additional and useless instances are started every 30 minutes in average.
This leads to high latency.

Since the big outage 1,5 month ago (http://code.google.com/p/googleappengine/issues/detail?id=8004)
it's the first time that my application slows down...

Hopefully Google does not need weeks again to fix the issue.

Cheers
Mos
 

Francois

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/8wQ-QQ9dA4AJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Francois Masurel]

It's getting worse :

If it's some kind of planned upgrade, shouldn't we have been informed about it?

Francois

[Vinny P]

It feels like Google rolled out an AppEngine update in the middle of the night, and something is going wrong. There are big latency spikes for many services right around the 4AM mark. 

-Vinny P

[Francois Masurel]

Yep it was far better 6 months ago :

http://code.google.com/status/appengine/detail/serving-java/2012/04/16#ae-trust-detail-helloworld-get-java-latency (80ms latency on average)

That's one big annoying thing you have to deal with when your app is hosted in the cloud and not on your own machines.

Hoping it will get fixed soon.

[Johan Euphrosine]

Please fill a new production issue on the public issue tracker with your application id:

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/u12V3uukCRMJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Francois Masurel]

Hi Johan, thanx for chiming in.

I think it's not related to my specific app but all java apps as it can be seen on the related status page :

http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency

Do you have some explanation about the progressive latency increase for the last 6 months?

These last few days, even a simple request making one simple datastore fetch takes an average of 200-300ms in my particular case.  These same requests were taking only 60-80ms a few months ago.

It looks like the GAE cluster is getting a bit overloaded, isn't it?

François

PS:

Production issue created: http://code.google.com/p/googleappengine/issues/detail?id=8292

[Johan Euphrosine]

Hi Francois,

I commented about the specific of your issue on the public issue tracker.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/2wmnq5Thm7oJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Brandon Thomson]

Yep, latency is way up in the last 4 hours or so on all three apps I'm monitoring, one Python, two Go. Very spiky. Hope someone's looking at it.

[Francois Masurel]

Hi Johan,

I have added a few appstats screenshots to the related issue : http://code.google.com/p/googleappengine/issues/detail?id=8292

Francois

[Francois Masurel]

Geez, the red light is on :

http://code.google.com/status/appengine/detail/serving-java/2012/10/16#ae-trust-detail-helloworld-get-java-latency

You can start betting about the max latency we will get today ;-)

Just kidding...  We are all behind you GAE team.  Get us a fix, quick.

Francois

On Tuesday, October 16, 2012 4:47:11 PM UTC+2, Francois Masurel wrote:

[Francois Masurel]

Things seems to have improved dramatically since a few hours, good job Google.

I can verify this in my logs too.

Will we ever know what happened exactly?

Francois