Google Workspace Status : Google Chat - AVAILABLE - 2025-11-27 10:00:50 UTC
2 views
Skip to first unread message
back...@gmail.com
Nov 27, 2025, 5:05:32 AMNov 27
to g-suite-st...@googlegroups.com
Product: Google Chat
Status: AVAILABLE
Published: 2025-11-27 10:00:50 UTC
# Incident Report
## Summary
On Monday, 24 November 2025, Google Chat experienced an overload which
led to service disruptions for users for a duration of 1 hour and 53
minutes. Customers encountered issues including the inability to send
messages, load the chat application, and start new conversations. Some
notifications were also incorrectly displayed with "Anonymous" as the
sender.
We deeply regret the disruption our Google Workspace customers
experienced. We are committed to improving our platform's reliability
and performance, and we are taking immediate steps to improve the
platform's performance and availability.
## Root Cause
The trigger of the outage was transient errors served from an internal
microservice in Europe that caused Chat backend processes to execute a
fallback path to retrieve this data. The root cause was that the data
retrieved from the fallback path subsequently caused an update to Chat
metadata which were sent to user devices, causing those devices to
reload data from Chat backend servers, thereby causing more updates to
be sent to user devices resulting in a cascading effect that
overloaded Chat servers in Europe.
Google engineers have implemented a mitigation to limit high rates of
requests to prevent this cascading loop from recurring.
## Remediation and Prevention
Google engineers were alerted to the incident via alerts and
immediately started an investigation.
To fully mitigate the impact, our engineering teams took the following steps:
1. **Increased Server Capacity:** To handle the unexpected surge in
traffic, we significantly increased the capacity of our servers in the
affected region to ensure the system could process all incoming
requests smoothly.
2. **Limited the Rate of Metadata Updates**: To prevent the cascading
loop of requests from overloading Chat servers again, Google engineers
have limited the rate of metadata updates so that Chat servers will
not be overloaded in this way.
3. **Paused Non-Essential Background Tasks:** To prioritize the
restoration of core chat functions, we temporarily halted several
internal background processes. This action helped to reduce the
overall load on our systems, freeing up resources to serve user
traffic more effectively.
4. **Gradual Traffic Restoration:** Our automated defenses had
initially blocked some traffic to prevent a complete system overload.
A crucial part of the recovery was to gradually and carefully allow
this traffic back. This controlled approach ensured that as users
reconnected, the system wouldn't be overwhelmed again, allowing
everyone to successfully re-establish their connection to Google Chat
without getting stuck in a retry loop.
Google is committed preventing a repeat of this issue in the future
and is completing the following actions:
* **Fix the immediate cause**: Stop errors in calls to the
microservices from triggering metadata updates to devices. The fix
prevents a cascading loop of client devices making requests back to
Chat servers resulting in overload.
* **Prevent devices from sending excessive load**: Implement changes
in Chat web and mobile apps to prevent them from performing full
reload of data from Chat servers when it is not needed. This reloading
contributed greatly to the overload of Chat servers.
* **Improve Monitoring**: Enhance monitoring to identify which
specific metadata updates to user devices cause reloads in order to
more quickly mitigate similar issues in the future.
## Detailed Description of Impact
On Monday, 24 November 2025 at 00:02 US/Pacific, Google Chat users in
Europe experienced a service disruption affecting core features of the
service for a duration of 1 hour and 53 minutes.
At its peak, the incident affected approximately 20.2% of users in
Europe. The issue was resolved by 01:55 US/Pacific.
Users encountered problems with the following core functionality:
* **Initial Application Load**: Many users had trouble loading the
Google Chat application.
* **Opening Chat Rooms**: Users experienced errors when trying to open
existing chat rooms.
* **Sending Messages**: Users faced delays sending messages.
* I**ncreased Latency**: There was an increase in message delivery
latency for affected users.
* **Notification Rendering**: Some notifications were also incorrectly
displayed with "Anonymous" as the sender.
Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/KZuQxLyBKvPD1zS6sxVA
Status: AVAILABLE
Published: 2025-11-27 10:00:50 UTC
# Incident Report
## Summary
On Monday, 24 November 2025, Google Chat experienced an overload which
led to service disruptions for users for a duration of 1 hour and 53
minutes. Customers encountered issues including the inability to send
messages, load the chat application, and start new conversations. Some
notifications were also incorrectly displayed with "Anonymous" as the
sender.
We deeply regret the disruption our Google Workspace customers
experienced. We are committed to improving our platform's reliability
and performance, and we are taking immediate steps to improve the
platform's performance and availability.
## Root Cause
The trigger of the outage was transient errors served from an internal
microservice in Europe that caused Chat backend processes to execute a
fallback path to retrieve this data. The root cause was that the data
retrieved from the fallback path subsequently caused an update to Chat
metadata which were sent to user devices, causing those devices to
reload data from Chat backend servers, thereby causing more updates to
be sent to user devices resulting in a cascading effect that
overloaded Chat servers in Europe.
Google engineers have implemented a mitigation to limit high rates of
requests to prevent this cascading loop from recurring.
## Remediation and Prevention
Google engineers were alerted to the incident via alerts and
immediately started an investigation.
To fully mitigate the impact, our engineering teams took the following steps:
1. **Increased Server Capacity:** To handle the unexpected surge in
traffic, we significantly increased the capacity of our servers in the
affected region to ensure the system could process all incoming
requests smoothly.
2. **Limited the Rate of Metadata Updates**: To prevent the cascading
loop of requests from overloading Chat servers again, Google engineers
have limited the rate of metadata updates so that Chat servers will
not be overloaded in this way.
3. **Paused Non-Essential Background Tasks:** To prioritize the
restoration of core chat functions, we temporarily halted several
internal background processes. This action helped to reduce the
overall load on our systems, freeing up resources to serve user
traffic more effectively.
4. **Gradual Traffic Restoration:** Our automated defenses had
initially blocked some traffic to prevent a complete system overload.
A crucial part of the recovery was to gradually and carefully allow
this traffic back. This controlled approach ensured that as users
reconnected, the system wouldn't be overwhelmed again, allowing
everyone to successfully re-establish their connection to Google Chat
without getting stuck in a retry loop.
Google is committed preventing a repeat of this issue in the future
and is completing the following actions:
* **Fix the immediate cause**: Stop errors in calls to the
microservices from triggering metadata updates to devices. The fix
prevents a cascading loop of client devices making requests back to
Chat servers resulting in overload.
* **Prevent devices from sending excessive load**: Implement changes
in Chat web and mobile apps to prevent them from performing full
reload of data from Chat servers when it is not needed. This reloading
contributed greatly to the overload of Chat servers.
* **Improve Monitoring**: Enhance monitoring to identify which
specific metadata updates to user devices cause reloads in order to
more quickly mitigate similar issues in the future.
## Detailed Description of Impact
On Monday, 24 November 2025 at 00:02 US/Pacific, Google Chat users in
Europe experienced a service disruption affecting core features of the
service for a duration of 1 hour and 53 minutes.
At its peak, the incident affected approximately 20.2% of users in
Europe. The issue was resolved by 01:55 US/Pacific.
Users encountered problems with the following core functionality:
* **Initial Application Load**: Many users had trouble loading the
Google Chat application.
* **Opening Chat Rooms**: Users experienced errors when trying to open
existing chat rooms.
* **Sending Messages**: Users faced delays sending messages.
* I**ncreased Latency**: There was an increase in message delivery
latency for affected users.
* **Notification Rendering**: Some notifications were also incorrectly
displayed with "Anonymous" as the sender.
Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/KZuQxLyBKvPD1zS6sxVA
Reply all
Reply to author
Forward
0 new messages