Google Workspace Status : Google Chat - AVAILABLE - 2025-11-25 18:28:13 UTC

[back...@gmail.com]

Product: Google Chat
Status: AVAILABLE
Published: 2025-11-25 18:28:13 UTC

# Preliminary Incident Report
We apologize for the inconvenience this service disruption/outage may
have caused. We would like to provide some information about this
incident below. Please note, this information is based on our best
knowledge at the time of posting and is subject to change as our
investigation continues. If you have experienced an impact outside of
what is listed below, please reach out to Google Workspace Support
using the help article https://support.google.com/a/answer/1047213.
## Summary
On Monday, 24 November 2025, Google Chat experienced an overload which
led to service disruptions for users for a duration of 1 hour and 53
minutes. Customers encountered issues including the inability to send
messages, load the chat application, and start new conversations. Some
notifications were also incorrectly displayed with "Anonymous" as the
sender
## Root Cause
The trigger of the outage was transient errors served from an internal
microservice in Europe that caused Chat backend processes to execute a
fallback path to retrieve this data. The root cause was that the data
retrieved from the fallback path subsequently caused an update to Chat
metadata which were sent to user devices, causing those devices to
reload data from Chat backend servers, thereby causing more updates to
be sent to user devices, resulting in a cascading effect that
overloaded Chat servers in Europe.
Google engineers have implemented a mitigation to limit high rates of
requests to prevent this cascading loop from recurring.
## Remediation
Google engineers were alerted to the incident via alerts and
immediately started an investigation.
To fully mitigate the impact, our engineering teams took the following steps:
1. Increased Server Capacity: To handle the unexpected surge in
traffic, we significantly increased the capacity of our servers in the
affected region to ensure the system could process all incoming
requests smoothly.
2. Limited the Rate of Metadata Updates: To prevent the cascading loop
of requests from overloading Chat servers again, Google engineers have
limited the rate of metadata updates so that Chat servers will not be
overloaded in this way.
3. Paused Non-Essential Background Tasks: To prioritize the
restoration of core chat functions, we temporarily halted several
internal background processes. This action helped to reduce the
overall load on our systems, freeing up resources to serve user
traffic more effectively.
4. Gradual Traffic Restoration: Our automated defenses had initially
blocked some traffic to prevent a complete system overload. A crucial
part of the recovery was to gradually and carefully allow this traffic
back. This controlled approach ensured that as users reconnected, the
system wouldn't be overwhelmed again, allowing everyone to
successfully re-establish their connection to Google Chat without
getting stuck in a retry loop.
## Detailed Description of Impact
For a duration of 1 hour and 53 minutes, Google Chat users in Europe
experienced a service disruption which manifested as an inability to
use core features of the service, such as sending messages and
starting conversations.

Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/KZuQxLyBKvPD1zS6sxVA