Product: Admin Console
Status: AVAILABLE
Published: 2026-01-23 02:07:21 UTC
# Incident Report
## Summary
Starting 19 January at 10:10, Google Workspace customers may have
experienced missing account metadata, including primary email address
and account creation time in the Admin Console. Additionally, some
customers experienced authentication failures for external services
via Single Sign-On (SSO). The issue lasted for 1 day, 1 hour and 39
minutes. To our Workspace customers whose services were impacted
during this disruption, we sincerely apologize. This is not the level
of quality and reliability we strive to offer you, and we are taking
immediate steps to improve the platform's performance and
availability.
## Root Cause
This incident was caused by corrupted user account metadata in a
secondary data store, triggered by a faulty code change.
In Google's infrastructure, user account information is maintained in
a primary backend system of record. This data is periodically
synchronized to secondary data stores, which applications such as the
Admin Console use to retrieve and display user profile information.
The trigger for this incident was a faulty code change deployed to a
synchronization job. This change caused the job to malfunction and
write invalid or empty information into the secondary data store for a
subset of user accounts. Although the correct user data remained
intact in the primary backend system of record, the corrupted copy in
the secondary data store was served to dependent services.
## Remediation and Prevention
Google engineers were first alerted to the potential issue via a
support case submitted on 20 January at 03:32. Upon receiving the
alert, engineering teams initiated an investigation and quickly
identified that a faulty code change, which had been recently deployed
to a synchronization job, was the primary source of the data
discrepancies. This specific job was responsible for updating
secondary data stores from the primary backend account system.
To resolve the incident, a multi-pronged remediation strategy was
executed. A rollback of the application service was performed in
parallel with a manual synchronization process to restore data for all
affected users. This restoration relied on the primary backend account
system, which remained accurate throughout the event. A specialized
script was developed and run to correct the corrupted metadata for the
impacted accounts. To ensure system stability during the recovery
phase, the script was deployed in controlled batches. This systematic
approach successfully mitigated the issue for all affected accounts
and fully restored services by 20 January at 11:49.
Google is committed to preventing a recurrence of this issue and is
implementing the following action items:
- Binary Release Controls: Automated release gates were established
for the affected synchronization service to ensure stability and
prevent a recurrence.
- Enhanced Release Validation: We are implementing stricter validation
checks within the synchronization job to detect and block invalid
metadata writes before they reach secondary stores.
- Monitoring and Alerting: We are expanding automated monitoring to
detect data integrity discrepancies between our primary system of
record and secondary data stores, allowing for faster detection.
## Detailed Description of Impact
The service disruption occurred between 19 January at 10:10 and 20
January at 11:49, lasting a total of 1 day, 1 hour and 39 minutes.
During this period, Google Workspace customers experienced significant
issues with missing account metadata, specifically impacting the
visibility and accuracy of user profiles.
- Admin Console Impact: Administrators searching for or viewing user
details found that primary email addresses appeared as empty fields.
Additionally, the account creation date for affected users was
incorrectly displayed as "January 1, 1970". This data corruption also
cascaded into other areas where user email addresses are typically
populated, such as Reporting logs and Activity events, hindering
administrative oversight and auditing.
- Directory API and Tooling: The issue extended beyond the web
interface to programmatic access. Applications utilizing the Directory
API, including critical tools like Google Cloud Directory Sync (GCDS),
received incomplete responses. These API responses were missing vital
fields, including primary emails and account creation metadata,
potentially disrupting automated synchronization and provisioning
workflows.
- Single Sign-On (SSO) Failures: A subset of customers experienced
authentication failures for external services. These failures occurred
when the external services relied on Google Workspace as the identity
provider via SSO, as the required user identifiers were unavailable
during the disruption.
Link to official detail of this status:
https://www.google.com/appsstatus/dashboard/incidents/9q5n4sfDnxYzHj8FxEua