Google Workspace Status : Gemini - AVAILABLE - 2025-09-30 05:12:58 UTC
0 views
Skip to first unread message
back...@gmail.com
Sep 30, 2025, 1:20:28 AM (2 days ago) Sep 30
to g-suite-st...@googlegroups.com
Product: Gemini
Status: AVAILABLE
Published: 2025-09-30 05:12:58 UTC
# Mini Incident Report
We apologize for the inconvenience this service disruption/outage may
have caused. We would like to provide some information about this
incident below. Please note, this information is based on our best
knowledge at the time of posting and is subject to change as our
investigation continues. If you have experienced impact outside of
what is listed below, please reach out to Google Cloud Support using
https://cloud.google.com/support or to Google Workspace Support using
help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
**Incident Start:** 29 September, 2025 10:53
**Incident End:** 29 September, 2025 11:42
**Duration:** 49 minutes
**GWS Affected Services and Features:**
Workspace Gemini and NotebookLM
**Regions/Zones:** Global
**Description:**
Various AI products experienced elevated 500 (internal) errors due to
the rollout of a faulty change to production. Google will complete a
full IR in the following days that will provide a full root cause.
**Customer Impact:**
Customers affected by this issue may have observed up to a 100% of
requests failing with '500' errors during the impacted time, between
10:53 and 11:42 US/Pacific on 29 September 2025.
**Additional details:**
A faulty change was rolled out to the control plane of the canonical
Machine Learning (ML) serving stack in one cluster. Traffic routers
that received their configuration from the faulty cluster assumed that
some models were deleted and returned errors to the clients. Due to
the error, the faulty control plane cluster had a significantly lower
load, which resulted in more routers getting their configuration from
the faulty cluster. This exacerbated the impact of the change.
The bad change was rolled back and all the errors returned by ML
serving were reduced to pre-fault levels, mitigating customer impact.
Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/6GdahVtGTryxTdh56iLP
Status: AVAILABLE
Published: 2025-09-30 05:12:58 UTC
# Mini Incident Report
We apologize for the inconvenience this service disruption/outage may
have caused. We would like to provide some information about this
incident below. Please note, this information is based on our best
knowledge at the time of posting and is subject to change as our
investigation continues. If you have experienced impact outside of
what is listed below, please reach out to Google Cloud Support using
https://cloud.google.com/support or to Google Workspace Support using
help article https://support.google.com/a/answer/1047213.
(All Times US/Pacific)
**Incident Start:** 29 September, 2025 10:53
**Incident End:** 29 September, 2025 11:42
**Duration:** 49 minutes
**GWS Affected Services and Features:**
Workspace Gemini and NotebookLM
**Regions/Zones:** Global
**Description:**
Various AI products experienced elevated 500 (internal) errors due to
the rollout of a faulty change to production. Google will complete a
full IR in the following days that will provide a full root cause.
**Customer Impact:**
Customers affected by this issue may have observed up to a 100% of
requests failing with '500' errors during the impacted time, between
10:53 and 11:42 US/Pacific on 29 September 2025.
**Additional details:**
A faulty change was rolled out to the control plane of the canonical
Machine Learning (ML) serving stack in one cluster. Traffic routers
that received their configuration from the faulty cluster assumed that
some models were deleted and returned errors to the clients. Due to
the error, the faulty control plane cluster had a significantly lower
load, which resulted in more routers getting their configuration from
the faulty cluster. This exacerbated the impact of the change.
The bad change was rolled back and all the errors returned by ML
serving were reduced to pre-fault levels, mitigating customer impact.
Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/6GdahVtGTryxTdh56iLP
Reply all
Reply to author
Forward
0 new messages