App Engine outage today

[Pete]

Hi all,

This morning at around 6:30am PDT we experienced a datastore outage
during which a small percentage of requests returned errors. Between
9:00 and 11:30am, and again at 12:40pm, the percentage of requests
returning errors increased significantly. At around 1:40pm we were
able to isolate the issue, and requests are currently serving
normally.

This outage was the result of a bug in our datastore servers and was
triggered by a particular class of queries. We have isolated the bug
and we're currently working on a fix. Going forward, we're also
working to further isolate queries so that in the future a bug like
this won't affect the stability of the system as a whole.

Thanks for being patient. We'll post further updates as we have them.

Pete, App Engine Team

[Pete]

Hi all,

Just an update on the issues we saw on Tuesday. We've identified the
root cause of the issue and implemented a fix. Specifically, we've
instituted a set of controls to ensure 1) that datastore queries no
longer trigger this particular bug and 2) that bugs like this in the
future don't affect the stability of the system as a whole. All of
our systems are currently operating smoothly, and have been since
1:40pm PDT (GMT-7) on Tuesday 6/17.

During this preview period, we on the App Engine team are working hard
to smooth out the system. At all times, though, we're trying to keep
system-wide outages like this to an absolute minimum so that your apps
will remain up and running. We're also trying to make sure that we
build effective ways to communicate with developers about the hiccups
that occasionally occur with large and complex systems like this, and
we'd welcome your feedback and ideas.

Pete Koomen, App Engine Team