Production issues, 2011-08-19

[Nick Johnson]

We are currently experiencing production issues with Google App Engine. Master/Slave applications may be experiencing performance degradation up to total unavailability. No disruption is expected for High Replication apps. The App Engine admin console is currently unavailable.

We are actively working on resolving the issue, and I will post updates as they are available. I sincerely apologise for the inconvenience this causes, and we're attempting to resolve the issue as soon as possible.

--
Nick Johnson, Developer Programs Engineer, App Engine

[Nick Johnson]

Update: Our production team continues to work on the issue. We will be briefly setting the Master/Slave datastore to read-only in order to facilitate restoring everything to normal.

-Nick

[Nick Johnson]

Update: The Masert/Slave datastore remains read only, but most M/S apps should now be serving correctly. The admin console is now functional for read-only reuqests.

-Nick

On Fri, Aug 19, 2011 at 12:10 PM, Nick Johnson <nickj...@google.com> wrote:

[Nick Johnson]

Update: Serving issues have now been resolved. All apps should be serving normally, though a small number of entity groups may be unavailable to read or write. These should be resolved over the next few hours.

Unfortunately, some master/slave datastore applications may experience rollback of transactions that were committed during the outage and in a short interval leading up to the outage. We will provide more updates on the details of this, and data recovery efforts, as it is resolved.

-Nick

On Fri, Aug 19, 2011 at 12:10 PM, Nick Johnson <nickj...@google.com> wrote:

[Nick Johnson]

Update: To the best of our knowledge, no data has been lost. A very small number of M/S apps may have transactions that were committed during the outage that are not visible. We will be contacting the administrators of those affected apps over the next few days to a week with more information. A postmortem with full details of the outage will be forthcoming.

-Nick

On Fri, Aug 19, 2011 at 12:10 PM, Nick Johnson <nickj...@google.com> wrote: