[fyi] socorro stackwalker now does code-info lookups for symbols

[William Kahn-Greene]

(cc:ing stability, crash-reporting-wg, and firefox-dev mailing lists)

I've been slogging through bug 1746940 for about two months now. I finally deployed the last part to production today, so you'll start noticing the effects of it.

In the past, if a crash report had modules where we had no debug file or debug id, but did have a code file and code id, then those modules would not get symbolicated. There was no way in our system for something to take a code file/code id and figure out the debug file/debug id in order to find the symbols file in the symbols bucket.

Bug 1746940 will fix that going forward. Now the Mozilla Symbols Server captures information in the header of the symbols file and stores it in the database. Additionally, the Socorro stackwalker will request symbols files using the code file/code id and the Mozilla Symbols Server will look that up and figure out the debug file/debug id (if it exists) and return that. This allows the stackwalker to symbolicate symbols in modules which we couldn't get symbols files for before.

You'll see evidence of this in a couple of places:

1. crash reports that used to have "xul.dll" or other unsymbolicated things in them will now have symbolicated things--we'll see changes in signature reports and maybe topcrashers

2. the cases of module/000000000000000000000000000000000 will drop in the missing symbols report

Skimming the last missing symbols report, I'm guessing this affects 10k out of like 300k crash reports. I'm not sure if those are all actionable or meaningful crash reports, but signatures like this (#3 in Top Crashers for Firefox 115.4.0esr):

OOM | large | mozalloc_abort | xul.dll | _PR_NativeRunThread | pr_root

switch to something like this:

OOM | large | mozalloc_abort | webrender::renderer::Renderer::render_impl

One caveat is that bBecause we have to capture information from the symbol file and store it in the database in the Mozilla Symbols Server, we won't have code file/code id -> debug file/debug id information for symbols files that were uploaded before October 2023. Let me know if you bump into these by writing up a bug in Tecken::Symbolication and I can backfill the information.

Hope this helps!

/will

--
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAKnh9qhHxOO-L-6aZLerJwkmysLgAhj5DrxTi4wQMhbvp1t%2B3Q%40mail.gmail.com.

[Jeff Muizelaar]

On Tue, Oct 24, 2023 at 2:40 PM William Kahn-Greene <wil...@mozilla.com> wrote:

(cc:ing stability, crash-reporting-wg, and firefox-dev mailing lists)

I've been slogging through bug 1746940 for about two months now. I finally deployed the last part to production today, so you'll start noticing the effects of it.

In the past, if a crash report had modules where we had no debug file or debug id, but did have a code file and code id, then those modules would not get symbolicated. There was no way in our system for something to take a code file/code id and figure out the debug file/debug id in order to find the symbols file in the symbols bucket.

Under what circumstances would we get a crash report that was missing the debug file/debug id for xul.dll?

 

-Jeff

--
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/CAKb2zs_0rop7ZsYdbiZjvwkkBjaghOQZ2W9vWio3WbBGOfip9g%40mail.gmail.com.

[Gabriele Svelto]

On 26/10/23 16:02, Jeff Muizelaar wrote:
> Under what circumstances would we get a crash report that was missing
> the debug file/debug id for xul.dll?

We don't know for sure, but most of the crashes which were missing those
bits of information were OOMs. I suspect that windbg.dll failed to
extract those values when memory was tight. This happened very often to
xul.dll in particular, so I wonder if the fact that it's a very large
library made this more likely.

Gabriele

--
You received this message because you are subscribed to the Google Groups "firef...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firefox-dev...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/firefox-dev/a81dac35-42ef-44b3-bb1f-5f4b2a921f1b%40mozilla.com.