Crash Reporting Headlines (September 2021)

[William Kahn-Greene]

(cross-posted to firefox-dev, stability, and crash-reporting-wg)

Status updates for crash reporting.

Quick Summary

• Made WER annotation and Mac memory pressure values data visible and searchable in Crash Stats

• Lots of improvements to dump_syms and symbol generation

Details

Completed

• Project: summary

• Crash stats: surfaced WindowsErrorReporting annotation

• https://bugzilla.mozilla.org/show_bug.cgi?id=1723691

• You can now search for crash reports that were caught with WER.

• Crash stats: flag windows crashes with OOM

• https://bugzilla.mozilla.org/show_bug.cgi?id=1723474

• Crash stats: expose Mac Memory pressure values

• https://bugzilla.mozilla.org/show_bug.cgi?id=1722626

• Crash reporter: catch more macOS exceptions

• Augmented the macOS exception handler to catch three more types of exceptions (EXC_RESOURCE, EXC_GUARD, EXC_CRASH)

• Modified both the Breakpad stackwalker and the oxidized one to correctly print them out

• https://bugzilla.mozilla.org/show_bug.cgi?id=1724368

• https://bugzilla.mozilla.org/show_bug.cgi?id=1724388

• https://bugzilla.mozilla.org/show_bug.cgi?id=1724928

• Crash reporter: cleaned up a bunch of old and useless crash annotations

• https://bugzilla.mozilla.org/show_bug.cgi?id=1723934

• dump_syms: produce proper CFI information for macOS builds

• Landed Alexis' changes to dump_syms in mozilla-central to produce proper CFI information for macOS builds]. Alexis did 100% of the work here while also documenting the previously undocumented Apple compact unwinding format.

• https://bugzilla.mozilla.org/show_bug.cgi?id=1723466

• https://gankra.github.io/blah/compact-unwinding/

• dump_syms: reduce duplicate symbols in builds

• Fixed a bug in symbolic that prevented it from printing out dynamic symbols in ELF executables. This caused dump_syms to emit duplicate symbols in our builds (instead of the correct ones) and no symbols at all in stripped builds (such as the ones in Arch Linux). This change is not in dump_syms yet as I'm waiting for Sentry to release a new version of Symbolic.

• https://github.com/getsentry/symbolic/issues/420

• dump_syms: reduce duplicate symbols in Andriod/ARM builds

• Fixed an issue that caused dump_syms to emit duplicate symbol entries in Android/ARM builds. It shrinks Fenix' libxul symbols by over 20%.

• https://github.com/mozilla/dump_syms/issues/263

• dump_syms: emit symbols for executable areas in ELF files that don't have one

• Made dump_syms emit symbols for executable areas in ELF files that aren't covered by one, yet. This improves the signatures when we crash inside the PLT or similar memory areas.

• https://github.com/mozilla/dump_syms/issues/264

• rust_minidump: small improvements for printing Linux exceptions

• symbols: fixed symbol scraper issues

• Should get us better symbols for recent builds in Ubuntu, Debian and Fedora.

In progress

• All: Rust rewrite of all things breakpad

• rust minidump-stackwalk:

• https://github.com/luser/rust-minidump/tree/master/minidump-stackwalk

• https://github.com/luser/rust-minidump/issues/153

• You can now install and test rust-minidump minidump-stackwalk

• Same CLI as existing minidump-stackwalk that Socorro uses. Outputs the same JSON schema.

• Work progresses.

• Tecken: new symbolication API microservice

• API url: https://symbolication.stage.mozaws.net/symbolicate/v5

• If you do any symbolication work, I'd love to know how it works for you and whether you encounter any issues.

• Work progresses on hardening the service. Will likely go to production in September 2021.

• https://bugzilla.mozilla.org/show_bug.cgi?id=1636210

• Firefox profiler: display richer symbol information and inline source view

• Display C++ and Rust source code with per-line sample counts right inside the profiler.

• Make the profiler display richer symbol information, specifically inline call stacks with file and line information.

• https://github.com/firefox-devtools/profiler/pull/3530

• Socorro: Better signature generation for Java crash reports (on hold)

• https://bugzilla.mozilla.org/show_bug.cgi?id=1541120

• There's a lot of design work that needs to be done here to figure out details. How should signature generation should work when we have Java and native stacks? How do we want to handle signature pieces when Java modules/classes have a lot of characters? Etc.

• It's not clear what the impact of this work will be. Will it help nightly triage? Will it help Fenix development? How can we measure that predicted impact?

• This work is on hold for now.

• Crash ping: Signature generation for crash pings (on hold)

• https://bugzilla.mozilla.org/show_bug.cgi?id=1631563

• Current status is in comment 16. It's blocked on getting the new symbolication API microservice into production and then overhauling the fx-crash-sig library.

• Symbols: improving process for acquiring symbols for macOS Big Sur

• https://bugzilla.mozilla.org/show_bug.cgi?id=1683758

• This enables profiles collected on beta versions of macOS with the Firefox profiler to have symbolicated system libraries and will improve stacks in crash reports for beta versions of macOS.

• Progress on this is ongoing.

More information

The crash reporting working group puts these out once a month so it's clearer what's going on in the wild world of crash reporting.

Wiki: https://wiki.mozilla.org/Data/WorkingGroups/CrashReporting

Matrix: #crashreporting

Mailing list: crash-reporting-wg

For support on symbols, stacks, stack unwinding, crash reports, Crash Stats, signature generation, and anything related to crash reporting, we hang out in #crashreporting on Matrix.

If this summary is missing something, please let us know!

/will

--
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAKnh9qiJHiiAV%3DWaLRC9Pns6t8CHOW8Qk-szd8sojrjtOC0e-g%40mail.gmail.com.