Google App Engine Issues on Wed April 9th 2014

[Google App Engine Downtime Notify]

We're investigating an issue with Google App Engine requests beginning at Wednesday, 2014-04-09 1:57 PM US/Pacific.

We will provide more information shortly.

[Google App Engine Downtime Notify]

The problem with Google App Engine was resolved as of Wednesday, 2014-04-09 13:06 (all times are in US/Pacific)]. We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

[Google App Engine Downtime Notify]

SUMMARY:

On Wednesday 9 April 2014, Google App Engine (GAE) experienced elevated error rates for some applications for a duration of 7 minutes. If your service or application was affected, we apologize - this is not the level of reliability and performance we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT:

On Wednesday 9 April 2014, some applications running in US datacenters with the High Replication Datastore experienced elevated error rates during the period 13:57 to 14:04 US/Pacific. 13.2% of applications that had more than 100 non-task queue requests during this period experienced an error rate of greater than 10%.

ROOT CAUSE:

The outage was caused by a software update which caused too many servers to restart simultaneously.

REMEDIATION AND PREVENTION:

Our System Reliability Engineering team has improved the system update process to ensure that we do not restart too many servers simultaneously.