Inlined function calls are now displayed in crash stacks

[Markus Stange]

Hi all,

I am happy to announce that crash reports just became a lot more useful!

We now recover inlined function calls and display them in the crash stack. In many cases, this allows you to identify the crashing code much faster than before. In the past, many crucial stack frames would have been omitted due to inlining. This made it borderline impossible, or at least extremely cumbersome, to find out where the crash actually happened.

Here's an example:

Before (inline calls lost): https://crash-stats.mozilla.org/report/index/ffddd880-b288-49af-bfd7-1dd640220901

After (inline calls recovered): https://crash-stats.mozilla.org/report/index/829c31ce-e2d8-4102-9188-d35fa0220908

In the "before" stack, you can see that the crash is inside TenuringTracer::collectToObjectFixedPoint(), but that's where the trail runs cold: The line number just shows you a call to traceObject, but you don't where inside traceObject the crash occurred.

In the "after" stack, the stack contains two more frames inside TenuringTracer::collectToObjectFixedPoint(): We now know about the inlined traceObject frame, along with the line number inside that function; and another frame is recovered: getDenseInitializedLength(). So the crash really happened in getDenseInitializedLength() all along.

This kind of multi-level inlining happens all the time, and it happens even more so in Rust code than in C++ code.

And now you can see right through it.

Supported versions:

The fix to emit inline information in symbol files landed last Friday in bug 1779631 and is now riding the 106 train. This means that, as of today, inline stacks are only available for crashes from Firefox Nightly.

Follow-up work:

Crash signatures do not include inline frames yet; this work will happen in bug 1788269.

Acknowledgements:

Most of this work happened in various github repos full of Rust code, specifically in mozilla/dump_syms, getsentry/symbolic and rust-minidump/rust-minidump. I'd like to thank all my reviewers for the fast reviews, especially Swatinem and loewenheim from Sentry, and gsvelto, Gankra and willkg from Mozilla.

Furthermore, I'd like to thank Gankra and willkg for implementing the final pieces to make the crash report infrastructure take advantage of the new symbol information.

It's also worth noting that I didn't have to touch any C++ code when doing this work. The Rust rewrites of dump_syms and rust-minidump have been in production for a few months now; if I had attempted this project before that, it would have involved a lot of C++ Google breakpad code and it would have been much less fun.

Background:

I worked on this project because symbol files are part of the shared infrastructure that's used both by the crash reporter and by the profiler. My main motivation was to get inline stacks into the profiler; we already had them for local builds, but not for "official" builds (i.e. builds for which symbol information comes from the symbol server). After doing all the necessary work for the profiler, making it work for crash reports was really straightforward and will surely pay off immensely.

Happy crash analysis!

-Markus

[Andrew Sutherland]

On 9/9/22 5:49 PM, Markus Stange wrote:
> Acknowledgements:

Many thanks to Markus and all involved in the ongoing effort to improve
crash-stats!  I expect this will help save me a lot of time and
significantly improve the quality of what I understand from
crash-reports in that reduced time!

Andrew

[Markus Stange]

On Sep 9, 2022 at 6:23:38 PM, Chris Peterson <cpet...@mozilla.com> wrote:

Very cool! I was comparing the two stack traces and did a double take when I saw the stack frame numbers repeating. But then I realized that the inlined functions wouldn't have their own stack frames. 🙂

Hah, yes, we went back and forth on whether to adjust the counting so that every inline frame gets its own frame index, or whether we just repeat the original frame number for each inline. In the end, repeating the frame index was simpler.

 Are inlined function calls also available in crash reports for Firefox Android and iOS?

Android: Yes, for crashes in our own native code (i.e. C++ / Rust etc). And the Java stacks have always been complete.

iOS: Probably not, I don't know how crash reporting works on iOS.

I know gsvelto is working on the oxidized minidump writer for Android. I don't know if your work depends on his work here.

The expansion of frame addresses into function names and inline stacks happens on the server side, when the minidump is processed, with the help of symbol files which also come from a server. So no extra information in the minidumps is needed, and this work is independent of minidump writing.

Cheers,

Markus

thanks,
chris

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/CAL2j%2B_MyEwBey-NRSD2A_BNq8MbfAr2Ko72T-MU0DS1ycwANBw%40mail.gmail.com.

[Bobby Holley]

We have all dreamed about this for years — thank you Markus and everyone involved for finally making it happen.

--

[Jeff Muizelaar]

Here's another good example of how this makes things better:
https://crash-stats.mozilla.org/report/index/3921dc62-a712-43a7-9b70-0ce970220514
vs.
https://crash-stats.mozilla.org/report/index/1fbb67cf-dec1-4729-96d5-061f10220913

In the first crash report we get an unwrap() happening someplace under
Renderer::update_texture_cache():
https://hg.mozilla.org/mozilla-central/file/466ab31800308d9a75db091c5cde3eda636cf77d/gfx/wr/webrender/src/renderer/mod.rs#l2423
but no real indication where.

In the second crash report with inline stacks we get copy_into_staging_buffer():
https://hg.mozilla.org/mozilla-central/file/c7037dbd2de37f147a445b6dae361671ef71896a/gfx/wr/webrender/src/renderer/upload.rs#l389
That's the expression that has the actual unwrap()

Thanks for making it happen!

-Jeff