Google App Engine issue with 502 errors from custom domains beginning approx. 23:00 US/Pacific on June 23, 2014

[Dave Hughes]

A number of customers are experiencing elevated 502 errors when accessing App Engine applications via custom domains.  We are investigating and we will provide more information shortly.

[Dave Hughes]

The problem with Google App Engine was resolved as of Tuesday, 2014-06-24 02:05 (all times are in US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better. We will provide a more detailed analysis of this incident once we have completed our internal investigation.

[Google App Engine Downtime Notify]

SUMMARY:
On Monday 23 June and Tuesday 24 June 2014, some requests to Google App Engine applications served from custom domains received 502 errors for a duration of 12 hours 28 minutes.  If your service or application was affected, we apologize — this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability.

DETAILED DESCRIPTION OF IMPACT:

On Monday 23 June 2014, from 13:23 until Tuesday 24 June 2014 01:51 US/Pacific, some requests to App Engine applications served from custom domains received 502 “Bad Gateway” errors.  During this period, 1.7% of custom domain requests received 502 errors and 16% of applications serving from custom domains were affected. Requests to appspot.com URLs were not affected by this issue.

ROOT CAUSE:

The issue was caused by an update to Google’s traffic management systems, which resulted in a misconfiguration in the system that maps custom domains to their corresponding App Engine applications.

REMEDIATION AND PREVENTION:

Google engineers corrected the faulty configuration to resolve the issue. In order to be able to resolve issues more quickly in future, we will improve our monitoring to quickly detect and alert for elevated error rates when serving custom domains.