Google Workspace Status : Gmail - AVAILABLE - 2026-02-09 05:03:28 UTC

[back...@gmail.com]

Product: Gmail
Status: AVAILABLE
Published: 2026-02-09 05:03:28 UTC

# Incident Report
## Summary
On Saturday, 24 January 2026, beginning at 05:02 US/Pacific, Gmail
experienced high failure rates in spam checking for a duration of 4
hours and 53 minutes. Affected users saw a banner indicating their
messages were not scanned for spam and saw some messages not labelled
accordingly as promotional and social.
Following recovery of the above-mentioned issues, some Gmail users
experienced delivery delays up to 10 minutes or less. This issue was
resolved for most users by 09:55 US/Pacific on Saturday, 24 January
2026.
We recognize the critical nature of Gmail's reliability to your
operations and extend our sincere apologies for any disruption this
incident may have caused to your organization. We are implementing
immediate measures to enhance system performance and stability to
prevent such an occurrence from happening again.
## Root Cause
This incident was caused by an overload of Gmail's spam checking
systems, which in turn was triggered by a temporary failure in a
backend service. This led to a cascading failure due to excessive
retries.
The sequence of events leading to customer impact was as follows:
* **Trigger:** The incident began with a temporary error in a backend
component within Gmail's mail processing pipeline, which is
responsible for analyzing incoming messages.
* **Cause:** While the initial error was temporary, the system's
response to it caused the impact. A retry mechanism, designed to
handle such transient failures, generated a massive and sustained
volume of retry traffic. This cascading retry storm overwhelmed the
capacity of multiple backend systems for several hours.
* **Effect:** The overload on the spam checking services and related
systems meant that there were mail delivery delays and not all
incoming mail was fully classified. To protect users from potentially
malicious content, Gmail displayed a warning banner on the affected
emails. This issue resulted in some Gmail users experiencing message
delivery delays, which were generally 10 minutes or less.
## Remediation and Prevention
Google engineers were alerted to the issue via automated monitoring at
05:12 US/Pacific on Saturday, 24 January 2026, and immediately began
an investigation. The investigation identified high error rates and
latency across the classification pipeline.
To mitigate the impact of the incident, Google engineers implemented
several measures focused on increasing system capacity and temporarily
reducing load.
The key mitigation actions taken were:
* **Increasing Capacity:** Engineers increased resources for the
affected services. This included general capacity increases for the
spam classification services and related backend components. One
specific action was a significant, emergency increase of resources for
a backend service responsible for abuse detection.
* **Reducing System Load:** To alleviate pressure on the systems,
engineers temporarily disabled certain functionalities to free up
headroom for the spam classification services to recover.
* **Tuning Retry Logic:** To address the cascading issues caused by
excessive retries, the team rolled out changes to the retry logic in
the mail delivery system.
Google is committed to preventing recurrence by implementing the
following action items:
* **Enhanced Emergency Controls :** We are enhancing our emergency
control mechanisms and accelerating mitigation protocols to facilitate
rapid recovery through temporary manual traffic tuning overrides
during similar service disruptions.
* **Improved Load shedding:** Implement criticality-aware load
shedding across the spam system to protect services and subsystems
during overload.
* **Tune Message Delivery Retries:** Optimize the retry logic in the
mail delivery system to avoid overwhelming downstream services.
* **Improved Deadline Propagation:** Ensure that service deadlines are
correctly propagated and honored across the entire spam classification
stack.
* **Reshape Processing Tasks:** Increase the memory and compute
resources allocated to individual anti-abuse systems to handle spikes
more gracefully.
## Detailed Description of Impact
On Saturday, 24 January 2026 from 05:02 to 09:55 US/Pacific, many
Gmail messages experienced elevated failure rates in spam checking.
Affected users saw a banner indicating their messages were not scanned
for spam and saw some messages not labelled accordingly as promotional
and social, and experienced delivery delays, generally up to 10
minutes.
**User Experience:**
* **Warning Banners:** Users received a banner on incoming messages
stating: _"Be careful with this message. Gmail hasn't scanned this
message for spam, unverified senders, or harmful software."_
* **Delivery Delays:** Impacted messages experienced delivery delays,
up to 10 minutes. However, there were no instances of lost emails or
erroneously delivered during this period.
* **Message Labeling:** Users may have noticed that promotional or
social labels were not correctly applied during the disruption.

Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/NNnDkY9CJ36annsfytjQ