Resolved: Glean client_id Regeneration Incident aka "FOG-Migration-measured new profiles inflated due to regenerated client_id"

[Chris H-C]

As part of the Instrumentation Consolidation Project (aka "FOG Migration") switching Firefox Desktop data collection from Legacy Telemetry to Glean, we discovered that unreported I/O failures could cause the Glean SDK to believe that it was the first run of a profile, generating a new client_id, even when it wasn't the first run of that profile. Near as we can tell[1], this has been going on for as far back as we have Glean data (775 day retention). This problem has now been mitigated, with the fix shipping in Glean SDK v66.2.0 with our first Firefox release of 2026: Firefox 146.

# What effect did that have on analyses and reporting of Glean-sent data?

Not too much for most analyses. The inflation in DAU was a small, but measurable, 0.7%. Over a six-month period, only about 6% of clients experienced at least one regeneration. The affected profiles come from every country, run every version of Firefox, and have no specific hardware or software or configuration that identifies them out of the general Firefox population[2].

If your analysis is related to new profiles, the effect is more significant since each regeneration looks like a new profile even if it doesn't look like a DAU. You'll see a sudden drop in Glean-measured new profile counts from Firefox 146 and onwards. They'll still be higher than Legacy-measured new profile counts[3], but lower than past years' Glean-measured new profile counts.

# Questions?

If you have questions, please feel free to reply to this e-mail or reach out on the #glean channel on Matrix[4] or Slack.

Thanks,
Chris H-C (:chutten)

[1]: Detecting this is difficult because there are several situations that appear mostly or exactly the same as this like profile cloning, profile resets, and (until the data deletion is processed) a data opt-out followed by an opt-in. Glean SDK v65.2.2 (shipped in Firefox 145) was the first version where we managed to directly instrument the fault in vivo.
[2]: This was actually part of the problem of figuring out the cause of this problem. There's _maybe_ a slight increase in likelihood if you're running older or less-capable hardware. Maybe.
[3]: We measure new profiles via smaller but more-frequently-sent pings in Glean than we do from Legacy Telemetry, so we expect Glean-measured counts to be higher.
[4]: https://chat.mozilla.org/#/room/#glean:mozilla.org