Crash Reporting Headlines (May 7th, 2021)

[William Kahn-Greene]

(cross-posted to firefox-dev, stability, and crash-reporting-wg)

Status updates for crash reporting.

Quick Summary

• WER support in crash reporter
• Work continues on WER support so that we get crash reports for situations we're currently not getting any. Main process support should be done. Content process support is in progress.
• Socorro's minidump-stackwalker improvements
• Socorro's minidump-stackwalker was improved to emit additional Windows and macOS information. You can see this in the minidump-stackwalk output in the crash report view of Crash Stats.
• rust-minidump progress is moving along
• Work towards replacing Socorro's minidump-stackwalker with rust-minidump is progressing very nicely.
• Crash Stats lets you search by major_version
• Crash Stats has an improved Extensions tab in the crash report view
  

Details

Completed

• Crash reporter: WER support
• Windows Error Reporting interception landed last month and can intercept all main process crashes we were previously missing. This includes __fastfail() crashes, catastrophic OOM crashes, weird DLL injections and very late shutdown crashes. It significantly increased nightly crash rate which is good! Content process support is being worked upon.
• Socorro: minidump-stackwalker improved Windows information
• minidump-stackwalker was improved to print out richer information for Windows including unloaded modules, authenticode signatures, __fastfail() crash reasons, and NTSTATUS errors.
• Socorro: minidump-stackwalker __crash_info support for macOS
• minidump-stackwalker was improved to find and emit __crash_info information for Apple-specific error messages.
• Thank you, Steven Michaud!
• Crash reporter: fixed OOM crash annotations
• Alexandre modified the way we handle out-of-memory crash annotations so
  that it will never be missing again.
• rust-minidump: taught rust-minidump to parse MISC_INFO_5 format
  
• Taught rust-minidump to parse the MISC_INFO_5 format (and wrote tests/printing machinery for all the previous formats)
  
• https://github.com/luser/rust-minidump/pull/137
• rust-minidump: upgraded minidump-processor unwinder
  
• Upgraded the minidump-processor unwinder -- can now unwind with frame-pointers and scanning on x86 and x64
• https://github.com/luser/rust-minidump/pull/145
• rust-minidump: upgraded cli to match dump_syms
  
• Upgraded the minidump-processor CLI frontend to match dump_syms, and taught it to generate a JSON version of its report (format is "whatever the layout of the current types are", to be iterated on over time)
  
• https://github.com/luser/rust-minidump/pull/151
• dump_syms: better support for Apple's compact unwinding
  
• Taught symbolic (and therefore dump_syms) how to dump Apple's Compact Unwinding (.__unwind_info) format into breakpad's format for x86/x64, as well as wrote up a very thorough description of the format (that is otherwise missing from llvm's implementation, which is the only existing documentation of the format). Ideally when this lands it will fix Bug 1691022 (x64 macos missing CFI on socorro).
  
• https://github.com/getsentry/symbolic/pull/372/
• Crash Stats: added last error value to crash report view.
• https://bugzilla.mozilla.org/show_bug.cgi?id=1692312
• Crash Stats: redid process type support
• Redid process type support--now "parent" is the value for parent process crash reports and we're phasing out "browser".
• This makes it a lot easier to search for parent crashes and aggregations on process type work now.
• https://bugzilla.mozilla.org/show_bug.cgi?id=1701357
• Crash Stats: iImproved Extensions tab in report view
  
• The Extensions tab now shows the extension name, whether it's a system extension, and its signed state.
  
• https://bugzilla.mozilla.org/show_bug.cgi?id=1629943
• https://bugzilla.mozilla.org/show_bug.cgi?id=974968
• Crash Stats: fixed Bugs API to support POST as well as GET
• https://bugzilla.mozilla.org/show_bug.cgi?id=1282707
• If there are other APIs that would benefit by having POST support, let me know.
  
• Crash Stats: added search by major_version
• Added a major_version field and the ability to search it. This works for all crash reports  submitted after April 25th.
• Now you can do searches like "major_version = 88" and "major_version >= 88 and major_version < 90"
  
• https://bugzilla.mozilla.org/show_bug.cgi?id=1111612
• https://bugzilla.mozilla.org/show_bug.cgi?id=1401517
• Crash Stats: all Super Search fields now have exists/does-not-exist filter
  
• https://bugzilla.mozilla.org/show_bug.cgi?id=973894
• Socorro: Added support for multiple processing pipeline rulesets
• The first non-default ruleset I wrote is "regenerate_signature" which just regenerates the crash signature. It takes 1/10 the time regular processing takes. I'll use this going forward to regenerate crash signatures after signature generation changes.
• We can use this infrastructure for additional processing as well. That's been something we've talked about over the years.
• https://bugzilla.mozilla.org/show_bug.cgi?id=1705469
• Siggen: Released socorro-siggen 1.0.6
• This includes signature generation changes made since 1.0.5 as well as some minor fixes.
• Presentation: Socorro Overview: 2021
• Converted Socorro Overview presentation done at Data Club into a blog post.
• https://bluesock.org/~willkg/blog/mozilla/socorro_overview_2021.html

In process

• All: Rust rewrite of all things breakpad
  
• https://github.com/getsentry/symbolic/issues/375
• Tecken: new symbolication API microservice
• API url: https://symbolication.stage.mozaws.net/symbolicate/v5
• If you do any symbolication work, I'd love to know how it works for you and whether you encounter any issues.
• https://bugzilla.mozilla.org/show_bug.cgi?id=1636210
• Socorro: Better signature generation for Java crash reports: https://bugzilla.mozilla.org/show_bug.cgi?id=1541120

More information

The crash reporting working group puts these out once a month so it's clearer what's going on in the wild world of crash reporting.

Wiki: https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting

Matrix: #crashreporting

Mailing list: crash-reporting-wg

For support on symbols, stacks, stack unwinding, crash reports, Crash Stats, signature generation, and anything related to crash reporting, we hang out in #crashreporting on Matrix.

If this summary is missing something, please let us know!

/will