crash reporting data dictionary

[William Kahn-Greene]

[cross posted: stability, crash-reporting-wg, dev-platform]

Hi!

I finished up all the groundwork I needed to do to finally get us to a point where we could have a Crash Reporting Data Dictionary. I threw together the last bits for a first pass today and pushed it to production just now. You can see it on the Crash Stats site:

https://crash-stats.mozilla.org/documentation/datadictionary/

It's modeled loosely after the Glean Dictionary but focuses specifically on crash reporting data.

What you can see there:

1. up-to-date list of crash annotations that are supported by Socorro, descriptions, required permissions, types, data review urls

2. up-to-date list of things in the processed crash report, source annotations, Super Search field name, required permissions, examples of data, data caveats / gotchas

Some examples:

annotation: https://crash-stats.mozilla.org/documentation/datadictionary/dataset/annotation/field/BackgroundTaskName

processed crash field: https://crash-stats.allizom.org/documentation/datadictionary/dataset/processed/field/xpcom_spin_event_loop_stack

Going forward, whenever we add support for a new crash annotation, it'll automatically get added to the data dictionary. Further, whenever we add a new field to Super Search, it'll get automatically added to the data dictionary.

This is a first pass and it's kind of ugly and while there are some additional things I'm mulling over doing, I'd love to get some initial feedback. Does it work for you? Does it answer questions you have or have had while using Crash Stats? Are there things you wish it covered that it doesn't? Write up a bug, send me an email, ping me on #crashreporting on Matrix.

This is covered in bug 1803558 which while it was created this morning has been in my queue of things to do for years.

I hope this helps!

/will

[Andrew Sutherland]

This is super amazing!  It's invaluable to know the space of annotation
data that is available, and in particular, I love the example data from
the last 7 days.  Thank you!

Is the example data something that could potentially be fetched in an
aggregate JSON so that searchfox could provide the example data in
"inlay hint" style (ex: what VS Code does for inferred types for rust
analyzer and clangd) in source listings where the
`CrashReporter::Annotation::*` symbols are found in the source? (This
would be an optional display thing, and for this specific case maybe it
would be more likely to end up in a forthcoming sidebar.)

For example, I see there's a call to CrashReport::AnnotateCrashReport at
https://searchfox.org/mozilla-central/rev/2fc2ccf960c2f7c419262ac7215715c5235948db/ipc/glue/ProtocolUtils.cpp#76-88
with the `CrashReporter::Annotation::IPCSystemError` symbol used, and we
could hang the example data off that symbol if there's a JSON dictionary
that has the `IPCSystemError` name from
https://crash-stats.mozilla.org/documentation/datadictionary/dataset/annotation/field/IPCSystemError
that we can mangle using a mozilla-central specific config file. Right
now I do see the example data is in
https://crash-stats.mozilla.org/documentation/datadictionary/dataset/processed/field/ipc_system_error
which links to the former, but ideally the information would be directly
coupled or available elsewhere in the JSON object so that searchfox
could do a straighforward lookup in either direction.

Searchfox could also add one or more context-menu options like "Go to
crash stats docs for IPCSystemError" or "Search crash stats for
IPCSystemError" or things like that on the symbol, although that's
separable from having JSON dictionary as long as there's a predictable
URL scheme searchfox can use from the symbol name[1].

Andrew

1: Disclaimer: The context-menu functionality is still in progress, but
I'm confident about being able to get to that functionality over my
holiday break as it's essential to complete the first round of automatic
diagramming so it can be exposed to users.

[William Kahn-Greene]

The processing step normalizes and validates annotation data. That data is then indexed in Elasticsearch and that's why there's example data for processed fields that are indexed, but not for the source annotation or fields that aren't indexed. I don't plan to add "last 7 days" type examples for the source annotation values, but I have been adding examples to the documentation for annotations. For example, Steven added example data when he added this annotation:

https://crash-stats.mozilla.org/documentation/datadictionary/dataset/annotation/field/MacMemoryPressureSysctl

There's no API for accessing data dictionary data at the moment, but it's probably not hard to build. If you could write up a bug with how you need it to work to support Searchfox, I can make it happen.

/will

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/0956e3ff-286e-f8b7-9f8e-1ca57f54838c%40asutherland.org.

[Chris H-C]

Stepping in to first say: This is fantastic work, willkg!

Next: :asuth, do you think there'd be benefits to having similar context-menu options for Glean metrics? Like on `build_displaylist_time`[1] we could point to the Glean Dictionary entry for it[2] and/or GLAM[3]?

:chutten

[1]: https://searchfox.org/mozilla-central/source/layout/base/nsLayoutUtils.cpp#3108

[2]: https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/metrics/paint_build_displaylist_time

[3]: https://glam.telemetry.mozilla.org/fog/probe/paint_build_displaylist_time/explore?aggType=avg&currentPage=1&ping_type=*

On Thu, Dec 1, 2022 at 8:28 PM Andrew Sutherland <asuth...@asutherland.org> wrote:

[Andrew Sutherland]

On 12/5/22 9:26 AM, Chris H-C wrote:
> Stepping in to first say: This is fantastic work, willkg!
>
> Next: :asuth, do you think there'd be benefits to having similar
> context-menu options for Glean metrics? Like on
> `build_displaylist_time`[1] we could point to the Glean Dictionary
> entry for it[2] and/or GLAM[3]?

Absolutely!  In particular, both seem like places people would want to
go in a single click.  It's probably also worth thinking about whether,
in the medium term, it's worth trying to bake any statistics or
trendlines into the searchfox HTML as an "inlay hint" to provide
immediate context to those reading the code. This could also include
accessible sparklines or other compact, static charts which might also
be integrated into any automatically generated control flow graphviz
diagrams[1].

A few questions:

- For a link like
https://glam.telemetry.mozilla.org/fog/probe/paint_build_displaylist_time/explore?aggType=avg&currentPage=1&ping_type=*
are the GET query arguments there something that will be correct across
all metric types?  Would it better to drop the GET query args?  I see
the GLEAN dictionary page uses
https://glam.telemetry.mozilla.org/fog/probe/paint_build_displaylist_time/explore?app_id=


- I see on the dictionary page
https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/metrics/paint_build_displaylist_time
that there is a link to the yaml definition point at
https://github.com/mozilla/gecko-dev/blob/ce78234f5e653a5d3916813ff990f053510227bc/gfx/metrics.yaml#L14.
Is there automation that already produces taskcluster artifacts that
searchfox could consume to be able to understand the metrics' yaml
config files, or that could be further instrumented to teach searchfox
things?  Alternately, I have a plan in
https://bugzilla.mozilla.org/show_bug.cgi?id=1800016 to help searchfox
understand YAML (and potentially JSON) files via a combination of JSON
schemas and overlay heuristics, and maybe that would be a more
appropriate technique for processing the files. Arguably not all tooling
needs to be able to produce a full AST of its config files and it may
not be worth the extra effort if a schema or schema and a hacky custom
searchfox mapping can do it for free.

Andrew

1: In https://bugzilla.mozilla.org/show_bug.cgi?id=1773165#c2 I've been
thinking about showing some source excerpts in the control-flow
diagrams.  Arguably a labeled sparkline-style diagram could be even more
useful in these cases.  Probably the key thing for this would be that
for accessibility purposes we'd want to make sure we have some kind of
automated detection of notable discontinuities in the charts or other
alerts and that these are explicitly surfaced to searchfox so that in
the accessible dual of the graphviz diagram we could create a summary
table which could list all of the metrics and prioritize and indicate
those which have alerts with explanations.  Such a table would be useful
to everyone, of course, I just want to make sure that anything I'm
proposing visually encoding has an alternate representation that's not
dependent on preattentive processing
(https://www.csc2.ncsu.edu/faculty/healey/PP/) or people having to
manually inspect the entirety of a results set to find the needle in the
haystack.

[Chris H-C]

You can drop the query params, for sure. GLAM'll populate them with defaults or the current values (if this isn't the first time the user's visited GLAM). One note is GLAM is presently behind LDAP. Not sure if searchfox has any best practices about linking to non-public resources.

As for automation that produces taskcluster artefacts, there is not any of that at present. However, there are plans for the future to get taskcluster to act a little like the github action[1] that's responsible for updating the pipeline to any new metrics or pings. And, if you wanted to build something, glean_parser[2] is not only available in-tree, it's also on pip. You can use that to parse the metrics and pings files in the tree, and then extract whatever meaning you might like. See toolkit/components/glean/build_scripts/glean_parser_ext for the glean parser extensions we've added to Firefox.

And, if you'd like a more real-time conversation to happen, you can find me and my teammates over here: https://chat.mozilla.org/#/room/#glean:mozilla.org

:chutten

[1]: https://mozilla.github.io/glean/book/user/adding-glean-to-your-project/enable-data-ingestion.html#validate-and-publish-metrics

[2]: https://github.com/mozilla/glean_parser/

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/bac173e9-39f1-f246-c6b5-8e2050a4befd%40asutherland.org.

[Andrew Sutherland]

On 12/5/22 1:26 PM, Chris H-C wrote:
> You can drop the query params, for sure. GLAM'll populate them with
> defaults or the current values (if this isn't the first time the
> user's visited GLAM). One note is GLAM is presently behind LDAP. Not
> sure if searchfox has any best practices about linking to non-public
> resources.

I'm planning to add a new settings page
(https://bugzilla.mozilla.org/show_bug.cgi?id=1703656) that will let
people opt-in or opt-out of various functionality, especially for cases
like the context-menu where users may prefer a minimal experience.  I'll
be sure to put GLAM behind a setting that will allow someone to indicate
they don't have LDAP and don't want to see the LDAP options.  (Searchfox
has no LDAP awareness at this point and would be dependent on the user
indicating whether they have access.)  I'd probably make all of this an
opt-out since it is useful and extremely specific to these symbols.

> As for automation that produces taskcluster artefacts, there is not
> any of that at present. However, there are plans for the future to get
> taskcluster to act a little like the github action[1] that's
> responsible for updating the pipeline to any new metrics or pings.
> And, if you wanted to build something, glean_parser[2] is not only
> available in-tree, it's also on pip. You can use that to parse the
> metrics and pings files in the tree, and then extract whatever meaning
> you might like. See
> toolkit/components/glean/build_scripts/glean_parser_ext for the glean
> parser extensions we've added to Firefox.

Thanks!  It looks like the magic definition mechanism comes from a
custom constructor in a custom loader at
https://github.com/mozilla/glean_parser/blob/4e2b16193703a6195f9d59e420a8a9fa3e7070ef/glean_parser/util.py#L114
so that probably suggests that the tree-sitter approach of
https://bugzilla.mozilla.org/show_bug.cgi?id=1800016 would probably be
the most pragmatic for my purposes at this time since searchfox's
interests are more about syntax highlighting than the higher level
semantics glean_parser derives.  Also, I see there are schemas at
https://github.com/mozilla/glean_parser/tree/4e2b16193703a6195f9d59e420a8a9fa3e7070ef/glean_parser/schemas
(woo!) which means that if the searchfox tooling for that is
schema-aware (which seems feasible), it could still do some neat stuff.

> And, if you'd like a more real-time conversation to happen, you can
> find me and my teammates over here:
> https://chat.mozilla.org/#/room/#glean:mozilla.org

Also thanks!  I'll definitely be sure to continue the conversation there
and/or on bugs (and cc you) as appropriate. Ideally I'll have a
proof-of-concept in the new calendar year that can serve as a basis for
additional discussion and iteration.

Andrew