Reaching the 72 CPU-minutes/minute limit

[hovu96]

I post this to the groups because the Google App Engine Support is neither answering my questions I posted to the request form (http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest) nor increasing the "CPU-minutes/minute" limit.

A few weeks ago (during the last Apple keynote) we redirected all users from our german news website (www.mactechnews.de) to the app engine site (mtn-live.de/ticker/). We expected about 1000 request per second but in the first 5 minutes we received 1500 request per second. Before this event we increased the billing dollars but we weren't aware of the "72 CPU-minutes/minute" limit. It is a very simple website that just fetches some entries from the db and renders them to the html. All other resources cached "public".

Here are some screenshots that shows the situation:

CPU Seconds / Second:

CPU Milliseconds / Second:

Errors / Second:

Instances:

Requests / Second:

http://my-upload.appspot.com/serve/requestspersec.png?key=AMIfv97QiWwhl2VOW0NOnUt5lfGhxShRZ5sE0FyZTJS3Mx_HPI7YE9h4ubsHaHlmeHsIMkM65AC29DpR7jDjyzMrJW6kwnZsyAVrmU0CMBt4-wxo59gDMQXsm48UclII89Km4A7wyQnme5QZ-oZn8fxh5CgyEk7b4Q

We think the problem is shown in "cpusecspersec.png": The number of CPU seconds per second is larger then 72 (it is about 100).

After so many 500 errors we decided to redirect to our old ticker server (which is hosted on windows 2008). Some requests contain this dashboard logs message:

Request was aborted after waiting too long to attempt to service your 

request. This may happen sporadically when the App Engine serving 

cluster is under unexpectedly high or uneven load. If you see this 

message frequently, please contact the App Engine team.

But we really want to use App Engine for that and are planning to implement more feature like chatting on that page or some other community features... but this isn't possible at all if even the simplistic basic page can't be handled properly.

So our questions are:

1) Could you please increase the per minute quota ("CPU-minutes/minute") for the app id "sy-mtn" ? (The think the next Apple keynote will take place in September so it would be great to increase the limit as soon as possible)

2) Do you see any other (per minute) quota that we could reach?

Thanks,

Robert

[Philip]

Hi Robert,

there is a undocumented feature called edge caching. If your handler
returns the Cache-Control header a local google datacenter will create
a copy of the output and serve it to clients directly. If a page is
edge cached you don't consume any cpu quota and only have to pay for
traffic. Maybe you can make use of that feature for your ticker, I
don't think that the usability would be harmed if you let google cache
the ticker for 10-20 seconds.

Regards
Philip

On Aug 26, 10:07 am, hovu96 <robert.fuj...@googlemail.com> wrote:
> I post this to the groups because the Google App Engine Support is neither
> answering my questions I posted to the request form

> (http://code.google.com/support/bin/request.py?contact_type=AppEngineC...)

> nor increasing the "CPU-minutes/minute" limit.
>
> A few weeks ago (during the last Apple keynote) we redirected all users from
> our german news website (www.mactechnews.de) to the app engine site
> (mtn-live.de/ticker/). We expected about 1000 request per second but in the
> first 5 minutes we received 1500 request per second. Before this event we
> increased the billing dollars but we weren't aware of the "72
> CPU-minutes/minute" limit. It is a very simple website that just fetches
> some entries from the db and renders them to the html. All other resources
> cached "public".
>
> Here are some screenshots that shows the situation:
>
> CPU Seconds / Second:
>

> <http://my-upload.appspot.com/serve/cpusecspersec.png?key=AMIfv96C4JpG...>
>
> CPU Milliseconds / Second:
>
> <http://my-upload.appspot.com/serve/millisecspersec.png?key=AMIfv96h8j...>
>
> Errors / Second:
>
> <http://my-upload.appspot.com/serve/errorspersec.png?key=AMIfv95IspnDt...>
>
> Instances:
>
> <http://my-upload.appspot.com/serve/instances.png?key=AMIfv95j7esCZrUN...>
>
> Requests / Second:http://my-upload.appspot.com/serve/requestspersec.png?key=AMIfv97QiWw...

[hovu96]

Hi Philip,

We're already using this caching feature for most of the resources (Images, css, scripts, ...). But the main HTML page is not cached because it can contains user specific information like the email address in case a user logged in. Yes, it could be implemented to cache a default version of the html page for users not logged in and the logged in version is available using another address without using the public caching feature. But this is not what I want because we're planning to implement different features that heavily personalizes the page and users should always be logged in. Caching is not an option in this scenario.

I don't know why google doesn't respond to my questions or just increases the limit (only for our app id). We're willing to pay for the increased cpu usage caused by our app but we cannot afford that our service becomes unavailable under heavy load. That's why we began to use GAE!

Robert

[Johan Euphrosine]

Hi Robert,

Sorry if your quota request got lost, I was unable to find anything
related to you app in the quota request history.

Anyway feel free to open a production issue through:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

And I will followup quickly.

Thanks in advance.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/0-169Q2Fd0UJ.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[hovu96]

Hi Johan,

I opened a production issue:

http://code.google.com/p/googleappengine/issues/detail?id=5705

The only response to my form post (http://code.google.com/support/bin/request.py?contact_type=AppEngineCPURequest) was a email message from Wesley who asked me to tell him more details about my app. This was about 9 Jun 2011. The message subject contains "#815242820". If you want I can send you the entire thread.

Thanks,

Robert

[hovu96]

Is there anyone else who ran into a problem like this?

If yes, has google increased the limit (per minute quota) of your app?