Google App Engine issue with UrlFetch service beginning at 9:55am US/Pacific on June 18, 2014

[Google App Engine Downtime Notify]

We're investigating an issue with Google App Engine URLFetch service beginning at 2014-06-18 9:55 (all times are in US/Pacific). We will provide more information shortly.

[Google App Engine Downtime Notify]

The problem with Google App Engine URL Fetch service was resolved as of Wednesday, 2014-06-18 15:00 (all times are in US/Pacific). We apologize for the inconvenience and thank you for your patience and continued support. Please rest assured that system reliability is a top priority at Google, and we are making continuous improvements to make our systems better.

[Google App Engine Downtime Notify]

SUMMARY:

On Wednesday 18 June 2014, some Google App Engine applications experienced elevated errors and latency from the URL Fetch API for a duration of 6 minutes.

DETAILED DESCRIPTION OF IMPACT:

On Wednesday 18 June 2014, some App Engine applications experienced elevated errors and latency from the URL Fetch API at the following times (all in US/Pacific): 09:56 - 09:58, 10:37 - 10:38, 12:47 - 12:49, and 13:07 - 13:08. The incident affected 13.7% of applications hosted in US datacenters that used the URL Fetch API at these times. The error rate caused by the incident was 5.8%. The increase in latency was 31% at the median during the incident periods. The latency increased by 9 times at the 90th percentile. URL Fetch calls to Google APIs (except Cloud Storage) and appspot.com URLs were not affected.

ROOT CAUSE:

The incident was caused by an issue with the system that supports outbound connections from Google servers.

REMEDIATION AND PREVENTION:

To prevent recurrence, we added additional capacity to this system on Thursday 19 June 2014 at 02:19 US/Pacific.