High CPU usage and how to improve performance

[Prem D]

My python app engine app is somewhat popular, mainly used as via api
from my iphone app.


Recently I am seeing high over usage charges.

The most used call already use 15 CPU hours , with stats:
3.7 calls/sec 26.15K requests in 12hrs 603 (0) [cpu]

HTTP/1.1" 200 1839 - "QandA/5.4 CFNetwork/485.12.7 Darwin/10.4.0"
"may.latest.nuvusanswers.appspot.com" ms=1490 cpu_ms=467 api_cpu_ms=0
cpm_usd=0.013200 pending_ms=1058 throttle_code=1
instance=00c61b117c8b0de1286e8d90c22f4d5b3897c8

This call just reads a memcache and returns the data. Thats it. Will
really like to know are memcache reads so expensive ? Am I using a lot
more memcache that I need ?

I have removed all other versions and have just one version running.
The number of instances now is

28 total 0.510 (Average QPS) 281.1 ms (Average Latency) 59.5 MBytes
(Average Memory)

As this is the most expensive call, I am first going after that. but I
am seeing most of my calls are now using a LOT of CPU. Will be great
if anyone can point me to things I can debug to find out the high
usage across the board.

Thank you for reading this !

[Prem D]

Did Google change the way they calculate resource usage recently ? I
am seeing a lot of billing (~ $500 per month) now. If yes, is there
some documentation we can check.

[Robert Kluin]

Hi,
What it looks like is happening is that you're requests are getting
stuck in the pending queue. In order to get more instances you need
to reduce your call latency. In your case this might just be
something happening in passing, since the actual request only took a
couple hundred ms.

> ms=1490 cpu_ms=467 api_cpu_ms=0
> cpm_usd=0.013200 throttle_code=1

The pending ms value (1058) is the time the request was waiting to be
serviced by an instance; as I recollect it gets added to the request
latency (the ms value). As a note, that request does not seem to make
any API calls. I don't recall what throttle_code 1 means, would be
great if these were documented somewhere
(http://code.google.com/p/googleappengine/issues/detail?id=4405),

Robert

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[Barry Hunter]

Have you tried AppStats?
http://code.google.com/appengine/docs/python/tools/appstats.html

[Prem D]

Almost the same usage is now triggering 10times the daily cost ! Not
sure what changed as far as billing.

Checking appstats and instances.

On Jul 19, 2:55 pm, Barry Hunter <barrybhun...@gmail.com> wrote:
> Have you tried AppStats?http://code.google.com/appengine/docs/python/tools/appstats.html

[Prem D]

Is this is because of the new pricing, I would rather move out of
Google app engine.

We have over 10 apps serving our iPhone apps. It will take us a lot of
effort to port this code to php/mysql but I guess if the pricing
changes like this, we cannot afford to pay such high prices.

I am not sure if there is a phone number or support chat. When I am
paying $1000 per month for a service, I would expect to talk to a real
person and share with him the issues I am facing.

[Vinuth Madinur]

Someone suggested this:

http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

Not sure if we can get better response there, but do try and update if it works.

Thanks,

Vinuth.

[Vinuth Madinur]

Last couple of days the latencies and responses for my python app has been unstable. Latencies increased more than 10 times and threw DeadlineExceeded exceptions. But the system status page is a joke. It kept showing things were normal or sometimes being investigated. Right now, it shows no sign of anything for those days.

[Johan Euphrosine]

Hi Vinuth,

Is this issue resolved now ? What is your application id ?

Feel free to open a Production Issue if this is directly affecting
your application operation:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Johan Euphrosine]

Hi premdeb,

Feel free to open a Production Issue for problem like this affecting
directly your application operation:
http://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

If you do so, please reply to this thread with the issue number, so we
can followup.

Hope that helps.

[Johan Euphrosine]

About the new pricing model, it will only be launched later this year,
as announced during Google I/O 2011:
"""
All of these changes will take effect later this year. We will be
providing our customers samples of their new bills before the new
pricing model is launched so that they will be able to see what will
be changing.
"""
http://googleappengine.blogspot.com/2011/05/year-ahead-for-google-app-engine.html

About 1:1 support, we are currently rolling a Trusted Testers Program
for Paid support, feel free to contact me directly if you are
interested.

Hope that helps.

On Thu, Jul 21, 2011 at 10:24 AM, observer247 <pre...@gmail.com> wrote:

[Vinuth Madinur]

Yes. Things are working fine now.

[Prem D]

Issue number is 5415

It has been more than a week and I am yet to get a useful response on
the ticket.

On Jul 21, 4:20 am, Johan Euphrosine <pro...@google.com> wrote:
>
>
> Feel free to open a Production Issue for problem like this affecting

> directly your application operation:http://code.google.com/p/googleappengine/issues/entry?template=Produc...

>
> If you do so, please reply to this thread with the issue number, so we
> can followup.
>
> Hope that helps.
>
>
>
>
>
>
>
>
>
> On Thu, Jul 21, 2011 at 10:24 AM,observer247<prem...@gmail.com> wrote:
> > Is this is because of the new pricing, I would rather move out of
> > Google app engine.
>
> > We have over 10 apps serving our iPhone apps. It will take us a lot of
> > effort to port this code to php/mysql but I guess if the pricing
> > changes like this, we cannot afford to pay such high prices.
>
> > I am not sure if there is a phone number or support chat. When I am
> > paying $1000 per month for a service, I would expect to talk to a real
> > person and share with him the issues I am facing.
>

> > On Jul 20, 1:14 am,observer247<prem...@gmail.com> wrote:
> >> Almost the same usage is now triggering 10times the daily cost ! Not
> >> sure what changed as far as billing.
>
> >> Checking appstats and instances.
>
> >> On Jul 19, 2:55 pm, Barry Hunter <barrybhun...@gmail.com> wrote:
>
> >> > Have you tried AppStats?http://code.google.com/appengine/docs/python/tools/appstats.html
>

> > For more options, visit this group athttp://groups.google.com/group/google-appengine?hl=en.

[Johan Euphrosine]

Sorry for the late followup, I just updated your issue.

FYI, we believe that the problems experienced by most of the Apps on
this thread were due to a temporary datastore latency, as you can see
there:
http://code.google.com/status/appengine/detail/datastore/2011/07/27#ae-trust-detail-datastore-get-latency

This cascaded into a higher HTTP latencies for some Python
applications running on Master/Slave Datastore Replication system:
http://code.google.com/status/appengine/detail/serving/2011/07/27#ae-trust-detail-helloworld-get-latency

We offered the affected app to assist them to migrate to HRD, where
apps are not subject to those transient latency issues.

For the record Issue #5415 might be a different beast, as appstats
doesn't report any datastore operation but still high latency.

I already escalated it to the engineering team and will followup on
the bug report once I have any updates.

Hope that helps.

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Stéphane Cohen]

Hi Johann,

I jump on the opportunity to be part of the trusted testers programm :

We are running 2 Marketplace apps hosted on GAE with about 1500 domains registered after 3 months (without promotion)

This application is really strategic for our customers because it manages personnal contacts of the whole organization.

Our main app is running and consuming high level of CPU resources permanently.

The problem is that the current GAE environment is really causing a lot of clash in the applications and a lot of customers unhappy that are asking refunds everyday because the app does nto work on a regular basis.

The main problem with encounter are deadlocks on datastore, impossibility to upload application sometimes etc..

We are paying nearly 600 USD each Week for GAE hosting and we have get no service at all from Google when the system is down or when the system doesn't work as expected :

Any hosting company charging 2400 USD/ month of hosting would provide customer service + SLA. so it would be really great if you could sneak me in into your trusted tester programm for GAE for Business.

Thanks in advance,

Stef.

[Prem D]

I would like to know clearly if the over-usage charges are:

expected for the kind of usage I have and the code we have ? or
a bug caused by some bug or issue with app engine.

Based on that, we will make a decision. Because a simple memcache read
and return was using high cpu numbers. We need that answer to decide
whether to:

optimize the code ? or
wait for Google to fix issues on their side ? or
not wait for an infinite time before we can get an answer while paying
$600 a month and move to another host ?

As days are flying (15 days gone by already), I am almost decided that
it was a bad idea to assume that it was ok to host my application on a
proprietary network hosted by a reliable company. Now I guess we
should spend the next 15 days trying to rewrite the code in php/mysql
and move to a more portable host.

We are still waiting for the answer.

On Aug 3, 1:27 am, Johan Euphrosine <pro...@google.com> wrote:
> Sorry for the late followup, I just updated your issue.
>
> FYI, we believe that the problems experienced by most of the Apps on
> this thread were due to a temporary datastore latency, as you can see

> there:http://code.google.com/status/appengine/detail/datastore/2011/07/27#a...

>
> This cascaded into a higher HTTP latencies for some Python

> applications running on Master/Slave Datastore Replication system:http://code.google.com/status/appengine/detail/serving/2011/07/27#ae-...

[David]

I've been down that road trying to get Google to fix their issues for
a good year. Almost all of my production issues filed have gone
unanswered. The ones that did were about a month later. I've come to
realize app engine is only good to get something up quickly or if you
don't care about service availability or support and you don't mind
not having any control when something behind the server API goes
wrong. Not to mention every time they push out new versions, things
break for up to a week before they start to look stable again. I've
already started porting my app to work on an environment I have more
control over when things go wrong. I'd suggest the same for anyone
that has constant issues like I have.

Good luck.

David

[Robert Kluin]

You might try to submit a billing issue asking for a credit or something:
http://code.google.com/support/bin/request.py?contact_type=AppEngineBillingSupport

I'd have documentation ready to demonstrate that the app is
experiencing abnormal latency leading to higher charges. Have you
tried profiling with AppStats? One thing I find very odd about your
log entry is that you're seeing 0 api cpu ms, yet you're making a
memcache call. Is that the case with all of your high-cpu requests?
Is it the case with all of your "normal" requests?

Robert

[Johan Euphrosine]

Sorry for the late followup,

Your issue has already been escalated internally on July 28 2011 but
single application failure usually take more time to get diagnosed
than problems affecting more than one application.

I just updated the production ticket with more detail on the ongoing
investigation.

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Johan Euphrosine]

Hi,

FYI, stats I computed for the weekly community update shows that since
3 weeks, all productions issues have been escalated in less than 48
hours, so hopefully this is getting better.

Also as part of the new pricing model going out later this year, we
will provide SLA and Paid support, see the following page for more
details:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Regards,

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Prem D]

Is there a phone number I can call.

We have discovered something weird here. I think these billing
details are flawed ! Seriously did not expect this from Google. I am
waiting for a genuine response for almost a month and you continue
billing us.

And now this experiment from us tells me that there must be something
weird that happened in your system that your CPU calculations are
messed up and keeps billing us even for :

"return 0" statements.

600 Avg CPU for just a single "return 0".

I value my money but I am stuck with app engine till we can port our
code somewhere else.

On Aug 9, 12:43 am, Johan Euphrosine <pro...@google.com> wrote:
> Hi,
>
> FYI, stats I computed for the weekly community update shows that since
> 3 weeks, all productions issues have been escalated in less than 48
> hours, so hopefully this is getting better.
>
> Also as part of the new pricing model going out later this year, we
> will provide SLA and Paid support, see the following page for more
> details:http://www.google.com/enterprise/cloud/appengine/pricing.html
>
> Regards,
>
>
>
>
>
>
>
>
>
> On Mon, Aug 8, 2011 at 10:07 PM, David <wordsmithg...@gmail.com> wrote:
> > I've been down that road trying to get Google to fix their issues for
> > a good year.  Almost all of my production issues filed have gone
> > unanswered.  The ones that did were about a month later.  I've come to
> > realize app engine is only good to get something up quickly or if you
> > don't care about service availability or support and you don't mind
> > not having any control when something behind the server API goes
> > wrong.  Not to mention every time they push out new versions, things
> > break for up to a week before they start to look stable again.  I've
> > already started porting my app to work on an environment I have more
> > control over when things go wrong.  I'd suggest the same for anyone
> > that has constant issues like I have.
>
> > Good luck.
>
> > David
>

[Johan Euphrosine]

Hi,

There is no phone support for AppEngine ATM, as I mentioned before we
will offer SLA and Paid support as part of the new pricing model later
this year.

Please followup on the Production ticket you opened:
http://code.google.com/p/googleappengine/issues/detail?id=5415

If you could provide a test url for the CPU spikes you reproduce
there, that would help use to put a better diagnostic on the issue.

Thanks in advance.

> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Prem D]

So this is what we did:

24 hours back we changed the code to:

class GetUserAudio(webapp.RequestHandler):
def get(self):
self.response.out.write("0")
return


We just return after writing 0.

/getAvatar
1.3 40.63K 600 (0) 9%


CPU Time $0.10/ CPU hour
46%
46% 122.93 of 268.50 CPU hours $11.65

So we are paying around $1 per day for a call which just returns and
does almost nothing.

It takes 600 cpu_ms for this call ?


2011-08-11 22:28:30.125 /getUserAudio?userid=11445171&hasaudio=1
200 315ms 676cpu_ms 0kb ndA/5.4 CFNetwork/485.13.9 Darwin/11.0.0

166.205.10.107 - - [11/Aug/2011:22:28:30 -0700] "GET /getUserAudio?
userid=11445171&hasaudio=1 HTTP/1.1" 200 125 - "ndA/5.4 CFNetwork/
485.13.9 Darwin/11.0.0" "may.latest.nuvusanswers.appspot.com" ms=315
cpu_ms=677 api_cpu_ms=0 cpm_usd=0.018842
instance=00c61b117c3aa282423e18cdf09af851cead373f


If we are charged at 600 cpu_ms for a return 0 function, there is
definitely something wrong here. This has started happening around a
month back. Every day we are paying $20 for these issues. Worst part
is we still did not get an answer about if this is legitimate or a bug
in billing.

Please confirm why.

On Aug 11, 4:16 pm, Johan Euphrosine <pro...@google.com> wrote:
> Hi,
>

> ...
>
> read more »