Crash Reporting Headlines (August, 2021)
William Kahn-Greene
(cross-posted to firefox-dev, stability, and crash-reporting-wg)
Status updates for crash reporting. NB: This covers July and August.
Quick Summary
Windows Error Reporting crash collection and macOS crash handler improvements
Denoting more OOM crash reports as OOMs in the crash signature
Details
Completed
Crash Stats: flagging additional crash reports as OOMs
bug 1716742: flag ERROR_COMMITTMENT_LEVEL as OOMs
bug 1723474: flag WER windows crashes with a reason set to STATUS_FATAL_MEMORY_EXHAUSTION or STATUS_NO_MEMORY as OOMs
Crash reporter: WER improvements
Windows Error Reporting is fully functioning across all processes (bug 1697895 and bug 1682518), it flags OOM crashes correctly (bug 1711418) and the reports have a special WindowsErrorReporting annotation that lets you tell them apart from the rest (bug 1703761). Capturing hangs has also been disabled (bug 1718226).
Crash reporter: macOS crash handler improvements
The macOS crash handler has been modernized and now properly reports 64-bit crashes (bug 1035892). Among other things this makes UAF crashes on arm64 macOS builds immediately obvious as the poison pattern will appear as the crashing address.
macOS crashes now have thread names correctly populated (bug 1658831)
An infamous main process crash while capturing the minidump of a child process has been fixed on macOS (bug 1723941).
Crash reporter: native thread names support for minidumps from Linux
Martin Sirringhaus implemented native thread names support in Linux minidumps (bug 1714465) in child process (main process crashes still rely on the old machinery)
In progress
All: Rust rewrite of all things breakpad
rust minidump-stackwalk:
https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk
You can now install and test rust-minidump minidump-stackwalk
Same CLI as existing minidump-stackwalk that Socorro uses. Outputs the same JSON schema.
Work progresses.
Tecken: new symbolication API microservice
API url: https://symbolication.stage.mozaws.net/symbolicate/v5
If you do any symbolication work, I'd love to know how it works for you and whether you encounter any issues.
Work progresses on hardening the service. Will likely go to production in September 2021.
Socorro: Better signature generation for Java crash reports (on hold)
This is blocked on impact analysis and design work to figure out the details of how it should work.
- Let us know how better signature generation for Java crash reports helps you in the bug comments.
This work is on hold for now.
Symbols: improving process for acquiring symbols for macOS Big Sur
This enables profiles collected on beta versions of macOS with the Firefox profiler to have symbolicated system libraries and will improve stacks in crash reports for beta versions of macOS.
Progress on this is ongoing.
More information
The crash reporting working group puts these out once a month so it's clearer what's going on in the wild world of crash reporting.
Wiki: https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting
Matrix: #crashreporting
Mailing list: crash-reporting-wg
For support on symbols, stacks, stack unwinding, crash reports, Crash Stats, signature generation, and anything related to crash reporting, we hang out in #crashreporting on Matrix.
If this summary is missing something, please let us know!
/will
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAKnh9qge4a3DrXj2r25ccL9MBsRUv_ZmmAv%3DTXK6XPBd-Wa7jA%40mail.gmail.com.