Elevated Datastore Latency causing Deadline Exceeded Errors

[App Engine Downtime Notify]

As of 10am PST, November 2nd, 2010, we begin seeing elevated datastore
latency rates. Higher latency then caused a spike in
DeadlineExceededErrors across all App Engine applications. We quickly
determined this latency was being introduce by an ongoing upgrading to
a component of out infrastructure. After the upgrade was completed,
latencies did improve, but elevated error rates are still present.

As of 2:30pm PST, we are still seeing higher than average latencies
resulting in occasional errors. Investigation is continuing into the
cause and we will continue to update progress here.

- App Engine Team

[Sean Lynch]

As of 4pm, latency levels have returned to normal. We are monitoring
the situation carefully to ensure they stay stable and will update
this thread if we determined the issue is not all clear.

[App Engine Team]

As of 6:15pm PST, we saw this issue resurface under the infrastructure
powering the App Engine Administration Console as well as a small
percentage of other App Engine applications. This was resolved again
by 7:00pm PST.