Latency Issue - More Feedback!

[Greg]

Please can we have more information about the current latency issue.
Pointing us to http://groups.google.com/group/google-appengine-downtime-notify
doesn't help when the last post there is two days old.

Specifically:
1. Cloudstatus.com shows very clearly that everything went to hell in
handbasket after the "upgrade" earlier this week. Can Google confirm
this is the cause of the latency?

2. Is the scheduled maintenence next week likely to cure the problem
(then please bring it forward) or make it worse (then please delay
it)?

I'm sure the lack of information is because the engineers who know
what is happening are so busy trying to resolve the issue that they
can't post. Or maybe they are all smoking joints while watching Lost,
snickering about all the poor muppets out there struggling to find
ways of streamlining their apps to make them work. Without feedback we
don't know...

This is severely denting our (paying) customers confidence in my app,
and likewise our confidence in Appengine. We spent yesterday wringing
the last ounce of performance out of our app, and if I don't hear any
more, today (it's morning in New Zealand) will be spent investigating
how big a deal it will be to migrate it to a dedicated mod_python/
PostgreSQL system.

Other appengine developers - please star this message so we get a
response from Google.

[Pete Koomen]

Hi Greg,

We're listening, and we feel your pain. The App Engine team has been
working very hard to resolve this and we'll post updates as soon as we
have them. I'll try to address what I can right now:

1. The latency issues are related to the planned maintenance on Monday
night. They were unexpected, and we are working on infrastructure
improvements to ensure this won't happens again. If you'd like more
data on serving and API latency and error rate, you can look at our
own live monitoring data here http://code.google.com/status/appengine

2) We expect a fix to be in place by EOD to fix the issues you are
currently seeing. We will keep you updated if this is not the case.
When these issues are fixed, you should only see a small difference
(possibly 10-20%) in the performance of your application as compared
to earlier this week. However, one way or the other, we will return
our previous configuration by Monday 3/9.

Thanks for your patience through this.

Pete, App Engine Team

On Mar 4, 12:18 pm, Greg <g.fawc...@gmail.com> wrote:
> Please can we have more information about the current latency issue.

> Pointing us tohttp://groups.google.com/group/google-appengine-downtime-notify

[Artem]

Again, why does the status http://code.google.com/status/appengine
show normal?

Artem

On Mar 4, 4:13 pm, Pete Koomen <pkoo...@google.com> wrote:
> Hi Greg,
>
> We're listening, and we feel your pain.  The App Engine team has been
> working very hard to resolve this and we'll post updates as soon as we
> have them.  I'll try to address what I can right now:
>
> 1. The latency issues are related to the planned maintenance on Monday
> night.  They were unexpected, and we are working on infrastructure
> improvements to ensure this won't happens again.  If you'd like more
> data on serving and API latency and error rate, you can look at our

> own live monitoring data herehttp://code.google.com/status/appengine

[GregF]

Hi Pete -

Thanks a million. We now know when we can expect an improvement, and a
resolution. In our case I can now see that migration is going to be
more pain than hanging in there.

Can I strongly suggest you post the same text on google-appengine-
downtime-notify. And can you also pass the message that customers
would like at least 3-hourly updates during issues like this to
wherever it'll have most effect.

Cheers!
Greg.

[Brett Slatkin]

Hi Artem,

On Wed, Mar 4, 2009 at 1:43 PM, Artem <p.a...@gmail.com> wrote:
>
> Again, why does the status http://code.google.com/status/appengine
> show normal?

Over the course of the day the Status Site has shown "anomaly"
multiple times, but it has not yet exceeded the "disruption"
threshold. We will be posting more information about the issues soon
to show that we did indeed have a disruption of service during this
time period. We will also adjust our tolerances so this will show up
as a disruption automatically in the future. Thanks for your patience,

-Brett

[Alex Popescu]

I've mentioned this thread in a thread related to the same problem
that seems to have got more traction and more feedback:

Datastore operations slow Options:
http://groups.google.com/group/google-appengine/browse_thread/thread/e99f4e6eee9c5a5c

If you have more specific information (from logs, profiling sessions,
etc), I think it would be great to post them to that thread so we all
be able to track this problem.

cheers,

./alex