Google Cloud Platform Status
unread,Nov 13, 2015, 3:22:28 PM11/13/15Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Sign in to report message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to google-appengine...@googlegroups.com
SUMMARY:
On Tuesday, 10 November 2015, outbound traffic going through one of our
European routers from both Google Compute Engine and Google App Engine
experienced high latency for a duration of 6h43m minutes. If your service
or application was affected, we apologize — this is not the level of
quality and reliability we strive to offer you, and we have taken and are
taking immediate steps to improve the platform’s performance and
availability.
DETAILED DESCRIPTION OF IMPACT:
On Tuesday, 10 November 2015 from 06:30 - 13:13 PST, a subset of outbound
traffic from Google Compute Engine VMs and Google App Engine instances
experienced high latency. The disruption to service was limited to
outbound traffic through one of our European routers, and caused at peak
40% of all traffic being routed through this device to be dropped. This
accounted for 1% of all Google Compute Engine traffic being routed from
EMEA and <0.05% of all traffic for Google App Engine.
ROOT CAUSE:
A network component failure in one of our European routers temporarily
reduced network capacity in the region causing network congestion for
traffic traversing this route. Although the issue was mitigated by changing
the traffic priority, the problem was only fully resolved when the affected
hardware was replaced.
REMEDIATION AND PREVENTION:
As soon as significant traffic congestion in the network path was detected,
at 09:10 PST, Google Engineers diverted a subset of traffic away from the
affected path. As this only slightly decreased the congestion, Google
Engineers made a change in traffic priority which fully mitigated the
problem by 13:13 PST time. The replacement of the faulty hardware resolved
the problem.
To address time to resolution, Google engineers have added appropriate
alerts to the monitoring of this type of router, so that similar congestion
events will be spotted significantly more quickly in future. Additionally,
Google engineers will ensure that capacity plans properly account for all
types of traffic in single device failures. Furthermore, Google engineers
will audit and augment capacity in this region to ensure sufficient
redundancy is available.