More than 1800 QPS

Hernan Liendo

unread,

Feb 20, 2012, 10:25:00 AM2/20/12

to Google App Engine

Hi there!,

six month ago we've decided to use AppEngine to host our games. After
a successful launch we are starting to worry about its limits.
We have a growing game with approximately 30.000 daily unique users
and this is just the beginning.

We are serving more than 40 instances with a 300ms latency and 1.800
QPS. The problem is that we're having 3 to 4 timeout errors per minute
in our log. Most of them are related to the cache get method.
Our cache statistics are 548.459.776 cache hit, 1.810.909 cache missed
having 5.305 objects living in it.

If we fail to get the object from the cache, we're still trying to get
it from the DataStore. However we always reach the 30secs timeout.

Do you guys have any suggestions? Are we missing anything? How can we
deal with this?

Thanks in advanced,
Hernan

Robert Kluin

unread,

Feb 21, 2012, 2:45:33 AM2/21/12

to google-a...@googlegroups.com

Hey Herman,
You're hitting the timeout on the get from memcache? One suggestion
would be to figure out what the typical response time is then set a
lower deadline, such as 5 seconds. That will let you catch the
deadline error on the cache request sooner and get the value from the
datastore. I've successfully used this method in high-qps situations
several times.

Also, the user-request deadline is now 60 seconds.
http://code.google.com/appengine/docs/python/runtime.html#Quotas_and_Limits

Congrats on getting into these types of issues. ;)

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>

Hernan Liendo

unread,

Feb 21, 2012, 8:10:30 AM2/21/12

to Google App Engine

Thanks Rober,

giving a deep look into huge logs files we found that one specific
entity (700 rows, no writing, a lot a reading) was causing problems.
We replaced it with a static variable (synchronized and null safe in
case a new server instance creation).

Also Johan has gave me a couple of good advices: use automatic-
automatic for the pendency latency settings and try appstat.

We have no MemCache problems since then. Happy ending :D

On Feb 21, 4:45 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hey Herman,
> You're hitting the timeout on the get from memcache? One suggestion
> would be to figure out what the typical response time is then set a
> lower deadline, such as 5 seconds. That will let you catch the
> deadline error on the cache request sooner and get the value from the
> datastore. I've successfully used this method in high-qps situations
> several times.
>
> Also, the user-request deadline is now 60 seconds.

> http://code.google.com/appengine/docs/python/runtime.html#Quotas_and_...