(cross-posted to firefox-dev, stability, and crash-reporting-wg)
Status updates for crash reporting.
Symbols: Eliot (new symbolication API service) is now in production!
You can access it at https://symbolication.services.mozilla.com/ .
While it is in production, there's still a handful of things to do like load test to verify scaling, documentation, and a better index page.
After that, I need to spend time migrating things from the old symbolication API to the new one.
This unblocks several big projects like symbolication and signature generation for crash pings, fixing symbolication for inline functions, etc.
Crash stats: added support for Windows 11
Crash stats: undo ShutDownKill fine-grained bucketing
A while back, we changed signature generation to breakup the ShutDownKill buckets into smaller buckets so we could look at individual issues. This created a bunch of issues one of which was that the TopCrashers list was overwhelmed with ShutDownKill signatures. While ShutDownKill issues are important, it's not something we're getting to, so for now we're going back to what we were doing previously where we have a single (actually two) ShutDownKill signatures.
Crash stats: make distribution id work in search and aggregations
The TelemetryEnvironment has a distributionId field which tells us what vendor built Firefox. We extracted this field and made it work with search and aggregations.
All: Crash Reporting Working Group All Hands
We're scheduling a Crash Reporting Working Group All Hands to cover status of current projects, what projects need, and future vision.
It will likely happen in October or November.
All: Rust rewrite of all things breakpad
You can now install and test rust-minidump minidump-stackwalk
Same CLI as existing minidump-stackwalk that Socorro uses. Outputs the same JSON schema.
Firefox profiler: display richer symbol information and inline source view
Display C++ and Rust source code with per-line sample counts right inside the profiler.
Make the profiler display richer symbol information, specifically inline call stacks with file and line information.
Socorro: Better signature generation for Java crash reports (on hold)
There's a lot of design work that needs to be done here to figure out details. How should signature generation should work when we have Java and native stacks? How do we want to handle signature pieces when Java modules/classes have a lot of characters? Etc.
It's not clear what the impact of this work will be. Will it help nightly triage? Will it help Fenix development? How can we measure that predicted impact?
This work is on hold for now.
Crash ping: Signature generation for crash pings (on hold)
Current status is in comment 16. It's blocked on getting the new symbolication API microservice into production and then overhauling the fx-crash-sig library.
Symbols: improving process for acquiring symbols for macOS Big Sur
This enables profiles collected on beta versions of macOS with the Firefox profiler to have symbolicated system libraries and will improve stacks in crash reports for beta versions of macOS.
Progress on this is ongoing.
The crash reporting working group puts these out once a month so it's clearer what's going on in the wild world of crash reporting.
Mailing list: crash-reporting-wg
For support on symbols, stacks, stack unwinding, crash reports, Crash Stats, signature generation, and anything related to crash reporting, we hang out in #crashreporting on Matrix.
If this summary is missing something, please let us know!