Latency problem still really a problem

[cz]

Our app is still experiencing at least double the latency as compared
to last week and the app is basically unusable. I'm really concerned
about this since the GAE team has stated that the latency issue has
been resolved.
This is so disappointing if the current performance is considered to
be fine. I'm really hoping that this isn't the case.

I'm curious as to how you are testing latency internally? Perhaps the
tests aren't reflecting very well some real world scenarios and that
some combination of datastore operations are particularly onerous.

In any case, latency is still much much worse than before the server
maintenance.

P.S. also getting random 502 errors which I assume is related to this
issue: http://groups.google.com/group/google-appengine/browse_thread/thread/391a6ecc2febfc22/f43d82010fb1b598

[Brandon Thomson]

They said it has been "stabilized." But I agree, it is still much
worse. The timeouts are the real problem; it's just impossible to
complete certain operations.

> issue:http://groups.google.com/group/google-appengine/browse_thread/thread/...

[Larry]

I've got stable performance in my application. It consistently runs 3
to 4 times slower than it used to. Apparently, application performance
should improve on Monday but in the mean time performance is quite
poor...

http://code.google.com/status/appengine

> issue:http://groups.google.com/group/google-appengine/browse_thread/thread/...

[WeatherPhilip]

On Mar 6, 7:14 pm, Brandon Thomson <gra...@gmail.com> wrote:
> They said it has been "stabilized." But I agree, it is still much
> worse. The timeouts are the real problem; it's just impossible to
> complete certain operations.

I get timeouts trying to put single entities into the datastore
(sometimes). These aren't big objects -- maybe a total size of 3k
bytes with 3 indexes. I also get timeouts just paging through result
sets -- i.e. when doing 'for (doc in doclist):' where doclist is
the result of filter operation.

It clearly is doing something when these timeouts happen as the CPU
time used is *much* higher than normal. Alternatively, maybe there is
a bug, and something goes into a hard loop that is only broken when
the timeout expires....... It would be nice to know whether the CPU
that we are being charged in these timeout cases is *real* CPU that
was actually consumed somewhere, or whether this is a notional CPU
charge for what "must have been happening -- but actually wasn't". If
the former, then I'd like to understand what was actually going on to
consume all of these CPU cycles.

Philip

[Tito]

We have been experiencing datastore timeouts and, even worse, 502
timeouts since last week. A few CPU intensive pages and queries that
were completing 100% of times until recently, now they constantly go
into "deadline exceeded error" (after 30 seconds, when it would take
only 1-2 seconds earlier) and return 502 errors.

Furthermore, GAE is sensibly slower on all requests. I was surprised
to see that on the status page, the google team says that "latency
issues stabilized". That is not what I have experienced.

On Mar 8, 6:07 pm, WeatherPhilip <philip-goo...@gladstonefamily.net>
wrote:

[Brandon Thomson]

I suspect that it is "real CPU" that is unfortunately not doing any
useful work. For example when you commit a transaction on a highly
contentious entity, you may need to retry 50 or more times before
succeeding. Each try wastes the same amount of CPU, but you only
succeed in committing the transaction once.

It's weird, I know.

As to why so much CPU is being wasted now is anybody's guess; Google
won't tell us what they changed.

On Mar 8, 1:07 pm, WeatherPhilip <philip-goo...@gladstonefamily.net>
wrote:

[herbert]

hi,
i'm wondering how bad your performance problems are - i'm new to
google app engine, just deployed my new app last weekend for the first
time.. and it was _always_ slow .. but i have no idea why ..

for example a page with maybe one datastore query needs 3274ms-cpu
according to the logs in the administration console ..

i have added the profiler which outputs: '11288 function calls (11000
primitive calls) in 0.401 CPU seconds' for the same request - so why
is app engine requiring 3.2 secs, while my app "only" takes 0.4secs
(for an almost static page this is still bad)

can anyone tell me if this is normal speed? what can i expect?


thanks,
herbert