Datastore latencies spikes ongoing September 14, 2010

[App Engine Downtime Notify]

We are experiencing continued occasional latency spikes in the
Datastore which underpins a number of App Engine systems. This
results in application errors, datastore query timeouts, taskqueue
execution delays, and occasional errors in the Admin Console. We are
working feverishly to reduce the impact of these short term while
fixing the cause of these spikes as quickly as possible. Please watch
this thread for further updates on our progress.

[App Engine Team]

Performance of the datastore has stabilized within tolerances for the
past 3 hours, but we're not yet confident the situation is completely
resolved. We are continuing to monitor the service closely and will
give another update within 12 hours.

[App Engine Team]

We have determined that this issue has been caused by changes made to
our production configuration during Monday's scheduled maintenance. We
are now cautiously optimistic we are in a more stable state, however,
if we aren't, we may need to roll back the changes made on Monday
which could potentially require the same one hour read-only period.
We will update this thread as the situation progresses.

[App Engine Team]

As of 10:30pm PDT, App Engine performance has been stable for the
previous twelve hours. At this point, we are more optimistic that the
oncall team's fixes have solved the known issues with the
configuration changes we made on Monday. However, we are still
diligently monitoring performance of the service for any regressions.
We will update this thread again within 12 hours.

On Sep 15, 10:27 am, App Engine Team <appengine.nore...@gmail.com>
wrote:

[App Engine Team]

As of 10:00am PDT today, we consider this issue resolved.

On Sep 15, 10:38 pm, App Engine Team <appengine.nore...@gmail.com>