exceptionally high latency, causing bad experiences and high hosting cost

[Eric Ka Ka Ng]

from around last 12 hours, we see an exceptionally high latency on some of our gae apps. before requests on average takes 200ms-1000ms. now it can takes 2000ms - 5000ms. I try to measure the time as local client. like make 10 requests, 1-2 times the latency is normal <1s, half of the time is like 2s-5s, sometimes it even takes 10-20s!! 

as a result, it also doubles the number of instances started by GAE, and so as the frontend cost. 

does anyone observe similar problem recently, or it's only me? is it google's infrastructure prb, or anything i can do? if it's google's infrastructure prb, i dont think it makes sense to transform such bad user experiences (high latency) into doubling our hosting cost....

- eric

[Daniel]

For the last 2-3 days I'm seeing the same thing.  I've gotten several mails from unhappy users that my game server is too slow.. I don't normally have complaints like this.   I'm also seeing Deadline Exceeded errors which I don't normally see.   In addition my server costs are over double as well, probably for the same reason. 

You're not alone.. do you have any leads from Google on this?

Daniel

[Daniel]

BTW, here's my milliseconds per request chart.  The problems started about 3 days ago and the data is dramatic.. here's a 7 day graph

[Rishi Arora]

If you guys think this may be because of high memcache API latency, please star this issue:
http://code.google.com/p/googleappengine/issues/detail?id=7554

Although it appears Google is being stubborn about it and has marked this issue "WontFix", blaming it all on Master/Slave datastore.  An issue that lasts this long can't be simply blamed on M/S.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/MJCF02agu5oJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Daniel]

I've got deadline exceeded errors cased when setting memcache.   I've stared the issue and attached my graph.  I'm on an HDR app, not M/S

To post to this group, send email to google-appengine@googlegroups.com.
To unsubscribe from this group, send email to google-appengine+unsubscribe@googlegroups.com.

[Rishi Arora]

I used appstats to show that initially, it appeared to be memcache GET calls causing high latency.  However, I changed my memcache calls to timeout after 1 second.  This is now causing my app to resort to datastore reads, and now its the datastore reads that are causing high latency.  See appstat chart below.

I'm curious to see what appstats will show for your HRD-based app.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/aV5lK9vm15YJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Hamilton]

I ran into a similar issue yesterday.  My issue was caused by the master/slave data store being deprecated.  I converted my application to the hrd data store 

https://developers.google.com/appengine/docs/adminconsole/migration 

http://code.google.com/p/googleappengine/issues/detail?id=7555 

and all is well now.  

Best Regards,

Hamilton