[PingCentre Replacement] [ACTION NEEDED] Migrating SQL analyses of Messaging System data. PingCentre-sent data will cease in Firefox 121
6 views
Skip to first unread message
Chris H-C
Nov 16, 2023, 5:47:36 PM11/16/23
to fx-da...@mozilla.org, data-org
Hello,
WHO:
All direct recipients (by Bcc) of this email have been identified as recent (in the past 4 months) users of PingCentre-sent Messaging System data. If you rely on Messaging System (including Onboarding) data queried via SQL, this is for you.
WHAT:
As previously communicated on October 13 to affected parties, the messaging_system.* namespace of datasets is now DEPRECATED and SHOULD NOT be used.
Instead you should use `firefox_desktop.onboarding` (for onboarding data) or `firefox_desktop.messaging_system` (for any Messaging System data (including onboarding if you prefer it in this shape)).
If you have Messaging System SQL analyses, they will require manual migration to keep working. Please follow these steps:
1) If you don’t need your analysis any more, stop using it and cancel any scheduled runs via re:dash. We have prepared a list of all analyses using PingCentre-sent Messaging System data in re:dash if you’d like a quick way to check: [1].
2) To migrate an analysis that’s still needed,
a) If it is an onboarding analysis, you should only need to replace “messaging_system.onboarding” with “firefox_desktop.onboarding” as the latter was constructed to mirror the former (with a couple of extra event_* columns for easier querying).
b) If it isn’t onboarding, then you will need to replace
FROM messaging_system.X_Y
with
FROM firefox_desktop.messaging_system WHERE metrics.string.messaging_system_ping_type = ‘X-Y’
(notice the change from underscore to hyphen).
In addition, you will need to use new column names in your SELECT. For instance, “message_id” is no longer a top-level column. You can find it in metrics.text.messaging_system_message_id[2]. Please consult the Glean Dictionary page for the “messaging-system” ping for the list of available columns (each metric’s page has a reference at the bottom for where in the `firefox_desktop.messaging_system` table it can be found): https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/pings/messaging-system
WHY:
This is to remedy data loss that was discovered in PingCentre-sent Messaging System data and to reduce the number of data collection systems in Firefox Desktop in accordance with Mozprod Shared Infrastructure OKR 3.7.
WHEN:
PingCentre-sent Messaging System data will cease with Firefox 121 which is scheduled to reach release on December 19, 2023. You can follow the implementation in https://bugzilla.mozilla.org/show_bug.cgi?id=1849006.
HELP:
If you need any help, or come across something weird while migrating, or even if you just want to chat about this, please reach out on Slack#pingcentre-migration.
:chutten
On behalf of the PingCentre Replacement Project
[1]: https://docs.google.com/spreadsheets/d/1yqXueoZWmLbAuy5Uk1qrJsSdfgtlelnDx6DXscsF3W8/edit#gid=713215968
[2]: https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/metrics/messaging_system_message_id
WHO:
All direct recipients (by Bcc) of this email have been identified as recent (in the past 4 months) users of PingCentre-sent Messaging System data. If you rely on Messaging System (including Onboarding) data queried via SQL, this is for you.
WHAT:
As previously communicated on October 13 to affected parties, the messaging_system.* namespace of datasets is now DEPRECATED and SHOULD NOT be used.
Instead you should use `firefox_desktop.onboarding` (for onboarding data) or `firefox_desktop.messaging_system` (for any Messaging System data (including onboarding if you prefer it in this shape)).
If you have Messaging System SQL analyses, they will require manual migration to keep working. Please follow these steps:
1) If you don’t need your analysis any more, stop using it and cancel any scheduled runs via re:dash. We have prepared a list of all analyses using PingCentre-sent Messaging System data in re:dash if you’d like a quick way to check: [1].
2) To migrate an analysis that’s still needed,
a) If it is an onboarding analysis, you should only need to replace “messaging_system.onboarding” with “firefox_desktop.onboarding” as the latter was constructed to mirror the former (with a couple of extra event_* columns for easier querying).
b) If it isn’t onboarding, then you will need to replace
FROM messaging_system.X_Y
with
FROM firefox_desktop.messaging_system WHERE metrics.string.messaging_system_ping_type = ‘X-Y’
(notice the change from underscore to hyphen).
In addition, you will need to use new column names in your SELECT. For instance, “message_id” is no longer a top-level column. You can find it in metrics.text.messaging_system_message_id[2]. Please consult the Glean Dictionary page for the “messaging-system” ping for the list of available columns (each metric’s page has a reference at the bottom for where in the `firefox_desktop.messaging_system` table it can be found): https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/pings/messaging-system
WHY:
This is to remedy data loss that was discovered in PingCentre-sent Messaging System data and to reduce the number of data collection systems in Firefox Desktop in accordance with Mozprod Shared Infrastructure OKR 3.7.
WHEN:
PingCentre-sent Messaging System data will cease with Firefox 121 which is scheduled to reach release on December 19, 2023. You can follow the implementation in https://bugzilla.mozilla.org/show_bug.cgi?id=1849006.
HELP:
If you need any help, or come across something weird while migrating, or even if you just want to chat about this, please reach out on Slack#pingcentre-migration.
:chutten
On behalf of the PingCentre Replacement Project
[1]: https://docs.google.com/spreadsheets/d/1yqXueoZWmLbAuy5Uk1qrJsSdfgtlelnDx6DXscsF3W8/edit#gid=713215968
[2]: https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/metrics/messaging_system_message_id
Reply all
Reply to author
Forward
0 new messages