Python 2.7 issues on April 9th

[Christina Ilvento]

On the evening of April 9th, some Python 2.7 apps had degraded service due to an isolation and resource allocation issue. This issue was caused by a Python 2.7 application consuming excessive capacity which impacted other applications due to a misconfiguration in our systems, which limited other Python 2.7 applications from creating new instances on demand as required by their traffic.

Although normally such an issue would be quickly identified and resolved, we recently made changes to improve efficiencies in our resource pools and our normal monitoring configurations were not pre-emptively changed to catch such misconfigurations. Upon detecting and diagnosing the issue, the application was isolated from other applications and Python 2.7 apps were able to create new instances normally.

We apologize for any inconvenience that this issue caused you or your customers. We take such events very seriously, and we have improved our global performance settings to guard against recurrence of this incident, and are improving our monitoring to provide increased alerting and visibility into the relevant performance metrics of the production environment.


Regards,
Christina Ilvento on behalf of the App Engine Team