Unscheduled App Engine Outage - February 24th, 2010

[App Engine Team]

Since 7:53am PST, App Engine has been experiencing an unexpected
outage affecting the majority of App Engine applications. The team is
working quickly to correct the cause and will have an ETA on the fix
shortly.

Please watch this thread for updates. We sincerely apologies for the
inconvenience.

[App Engine Team]

We are still actively working on the on-going outage. We've also
experienced a problem with our backup datacenter. We will continue to
provide status updates on this thread every thirty minutes.

On Feb 24, 8:36 am, App Engine Team <appengine.nore...@gmail.com>
wrote:

[App Engine Team]

As of 9:48am, all applications should now be available in read-only
mode.

On Feb 24, 9:30 am, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 10:09am, all application should be serving traffic and
successfully reading/writing from the datastore. Again, apologies for
the inconvenience of the outage. We will follow up with a post-
mortem.

On Feb 24, 9:50 am, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 1:00pm PST, we are seeing lingering write issues with a small
percentage of datastore entity groups. As well, a small percentage of
cron jobs are not running correctly. We believe both issues are
linked to the unforeseen outage earlier today and we are treating them
as high priority. We will update this thread hourly with more
information going forward.

On Feb 24, 10:19 am, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 2:00pm PST, we are continuing to investigate the lingering
datastore and cron issues. We will continue to update this thread as
we progress.

On Feb 24, 1:02 pm, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 3:00pm PST, we have determined the datastore writes and only
reads within transactions are affected. We are working on a fix for
the issue.

On Feb 24, 2:13 pm, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 6:00pm PST, this issue has been limited to a very small set of
existing entity groups. This thread will be updated once we have made
the fix.

On Feb 24, 3:01 pm, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

Note that at 3pm, we disabled billing on all App Engine applications.
We will also not be charging for any of today's usage before the
outage.

On Feb 24, 6:20 pm, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

As of 11:00 a.m. PST, billing should be re-enabled and functioning
normally for all applications.

On Feb 24, 7:18 pm, App Engine Team <appengine.nore...@gmail.com>

[App Engine Team]

We continue to work diligently in the wake of yesterday's unforeseen
App Engine outage on a number of issues, so we wanted to provide you
an update with our current list of tasks.

- Applications will not be charged for any App Engine resource usage
that occurred during Feb 24th, 2010.

- We are still working on fix for writes and transactional reads
affecting a small number of datastore entity groups. We have
determined that approximately 25 application IDs have been affected by
this issue. If you feel you are having difficulties with certain
entity groups in your application, please respond to this thread with
your App ID.

- We are working hard on putting together a detailed post mortem that
we will be making public. We expect to have this available to you
next week.

Again we sincerely apologize for yesterday's service disruption. We
take App Engine's reliability very seriously and we are confident we
can use the hard lessons learned from yesterday's event to improve our
offering to you.

On Feb 25, 12:33 pm, App Engine Team <appengine.nore...@gmail.com>