Consolidating decision making in mobile products
Alessio Placitelli
Dear all,
Thanks to a joint team effort started in H1 2022, we are now ready to consolidate mobile KPI and Search data on one single consistent telemetry system.
tl;dr - In the past, search and KPI data came from a mix of telemetry systems across all mobile products. Starting from January 1st 2023, we will use one single telemetry system consistently across all mobile products to gather critical search and KPI data. Collection of data using Glean has been in place since 2020 and a comprehensive comparative analysis of legacy vs Glean data has been conducted in support of this consolidation effort. Meanwhile we will continue to collect data via legacy systems until further notice.
Who should care?
Data practitioners within the Data Org and Product Owners in other orgs.
Affected tables
unified_metrics,
active_users_aggregates,
search.mobile_search_aggregates_for_searchreport,
search.mobile_search_aggregates,
search.mobile_search_clients_daily,
search.mobile_search_clients_last_seen,
search.desktop_mobile_search_clients_monthly,
search.search_revenue_levers_daily.
search_derived.mobile_search_aggregates_v1,
search_derived.mobile_search_aggregates_for_searchreport_v1,
search_derived.mobile_search_clients_daily_v1,
search_derived.mobile_search_clients_last_seen_v1,
search_derived.desktop_mobile_search_clients_monthly_v1,
search_derived.search_revenue_levers_daily_v1,
Affected products
Firefox iOS, Focus/Klar iOS, Focus/Klar Android
Why is this happening?
Our mobile products use a mix of data collection systems (some deprecated, some unmaintained, some broken). Data collected by these systems is used inconsistently for tasks/decisions on KPI and search data (e.g. data from one system is compared to data from a different system in a different product), leading to potentially inconsistent conclusions.
What happens now?
On April 3rd, 2023 we will change the views mentioned in the previous sections to only report Glean data starting from the cutover data. Please reach out to your Data Org partner for any question you may have on the impact on your work.
Topline findings of the validation phase
Regarding KPIs, we find that Glean and Legacy telemetry data generally report a similar number of rows for active users across a range of metrics, including app version, activity segment, release channel, and country.
With respect to search, user-facing view tables have all other search metrics except “sap” search counts already in Glean (and they are not available in Legacy). Legacy telemetry has been overcounting the “sap” search counts in Focus/Klar Android. In Focus iOS, searches in Glean were also including URLs being opened, which is now fixed.
References
Decision Brief - Consolidating data collection in mobile browsers
Proposal: Handling of Glean Data for Apps Yet to Start Using Glean for Primary Reporting
unified_metrics: Comparing Glean and Legacy Telemetry data for Focus on Android
Comparison: Glean and Legacy Search Metrics for Firefox iOS
Glean and Legacy Comparison for Focus iOS/Klar iOS
Glean and Legacy Comparison For Focus/Klar Android
Your friendly Data Org
George, Joy, Xuan, Alessio