High Replication - Large CPU spikes without Datastore Access

[Simon Knott]

I've already posted this on the Java GAE group, but maybe someone has
seen this issue on a Python-hosted app.

I'm carrying out some testing of my app against the HR-store, instead
of the M/S datastore, and I've noticed that occasionally I'll get a
request which has a massive CPU spike. These spikes consume
~30-40seconds of CPU time, when my usual request times are ~500ms.
I'd assumed that something must be fundamentally wrong with my
application against the HR datastore, but I've now had three requests
to AppStats which have the same problem and one of them was an
image!

For example:

/appstats/static/app_engine_logo_sm.gif 200 131ms 31811cpu_ms
<--- Image?!
/appstats/stats 200 536ms 32375cpu_ms
/appstats/stats 200 169ms 23878cpu_ms


Yesterday I had the request below in my log. It didn't touch the
datastore, had 6 calls to MemCache and according to AppStats had a
grand total time of 125ms.

/user/get 200 156ms 42020cpu_ms

What could cause these large CPU spikes? They aren't loading
requests, as I'm correctly getting those marked in my logs and they
are in the usual region of 2-3s duration. I also don't experience them
on the same app against the M/S datastore. My initial thoughts were
that AppStats was potentially the cause of the problem, until I saw
the request for a static image with massive CPU usage!

[Wim den Ouden]

I'm tuning for the hr.
update entities are fast
entities with 3 properties add, delete 149ms 101cpu_ms 84api_cpu_ms
(tested over more days)
entities with 25 properties
http://code.google.com/p/relat/source/browse/trunk/relat/e-comm/contactmodel.py
add, delete 1825cpu_ms 1807api_cpu_ms (tested over more days)
it also depend on what properties
http://code.google.com/p/relat/source/browse/trunk/relat/e-comm/journmodel.py
2160cpu_ms 2142api_cpu_ms (tested over more days)
i have taken out the properties which are not necessary at the moment.
(into new kinds)
All cud actions are done by taskqueque
gr
wim

2011/2/8 Simon Knott <knott...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

--
gr
Wim den Ouden
Custom applications, based on Google App Engine https://e-comm1.appspot.com/
Free open source E-commerce/E-bookkeeping/E-business framework (web)
apps, http://code.google.com/p/relat/

[Simon Knott]

Just to emphasise, I'm seeing these spikes on requests which don't hit the datastore at all, and I don't see the same spikes on the same app with the datastore set to M/S.

[Wim den Ouden]

Hi Simon,
I'm glad your mention the subject, to get a spead gain i moved out all
the transaction surrounding where possible before testing the hr
datastore.
But it seems optimized for transactions because it is much faster in a
transaction. After putting all put actions in a transaction the kind
which gives me the biggest spikes (journ) is now 755cpu_ms
718api_cpu_ms (1/3 remaining cpu_ms within a transaction????, without
a transaction is much, much more cpu_ms intensive) wow i'm glad. Gona
change everything back!!! (journ has about 10 indexes that is also
part of the speed difference)
I'm using python with jquery on the client, so a lot cpu usage is on
the client, i don't see other difference in use (spikes)
gr
wim

2011/2/8 Simon Knott <knott...@gmail.com>:

> Just to emphasise, I'm seeing these spikes on requests which don't hit the
> datastore at all, and I don't see the same spikes on the same app with the
> datastore set to M/S.
>

[master outside]

I has nothing to do with your code as you do not write the appstats interface.

My only guess is that when the Master DB is having problems it those with HR take more cpu time to get requests done. Did you see a spike while they were doing maintenance?

[Simon Knott]

I've been getting these spikes since I started testing on the 3rd Feb.  

As I'm just testing to see if my app works on the HR datastore, at the moment, there is very low traffic.  These spikes seem to occur when I haven't hit the app for a few hours, so I'd almost put them down to loading requests if they weren't so high and they were being logged as such.  I've started checking in the admin console before I hit the application and I always have two instances apparently waiting for requests.  On Sunday I hit a 10th of my free CPU allocation through 20 requests hitting my app, purely because the CPU spikes were so high.