Documentation on how to read crash reports

[Nicholas Nethercote]

Hi,

Do we have documentation on how to read crash reports? The only thing
I have found is this page:

https://support.mozilla.org/en-US/kb/helping-crashes

which is not bad, but is lacking some details. I suspect there is
quite a bit of crash report interpretation wisdom that is in various
people's head, but not written down anywhere...

Nick

[Nicholas Nethercote]

I also found this:
https://support.mozilla.org/en-US/kb/contributors-crash-course

Which is similar to the first one, though presented in more of a
tutorial style, and lacking screenshots. Both of them describe the
meaning of individual fields, which would be better to have in
crash-stats itself, so I filed bug 1275799.

[Benjamin Smedberg]

Here is a selection of docs that we've written over the years. Many are
incomplete.

https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Crash_reporting
https://developer.mozilla.org/en-US/docs/Crash_Data_Analysis
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump

--BDS


On Wed, May 25, 2016 at 2:06 AM, Nicholas Nethercote <n.neth...@gmail.com

> wrote:

> Hi,
>
> Do we have documentation on how to read crash reports? The only thing
> I have found is this page:
>
> https://support.mozilla.org/en-US/kb/helping-crashes
>
> which is not bad, but is lacking some details. I suspect there is
> quite a bit of crash report interpretation wisdom that is in various
> people's head, but not written down anywhere...
>
> Nick

> _______________________________________________
> dev-platform mailing list
> dev-pl...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>

[Benoit Girard]

There's some information I've learned about reading crash reports, which is
obvious now but wasn't when I was an intern many years ago, that isn't
really covered by these.

Here's my workflow when looking at crashes:
- Windows tells you if the exception occurred during a write or read. Look
at the exception type. This is often the first thing I look at and
important to keep in mind for the next step.
- Look at the crashing source first, check the blame here and up the stack
as well for a recent change.
- Correlate the crash start date with the blame dates in the crashing stack
and nearby. A strong match points to a regression that's easy to deal with
if identified quickly.
- Crash address can give you a hint as to what's going on: 0x0 -> null
crash, 0x8 -> null offset accessing a member like this->mFoo, 0xfffffffffd8
-> accessing stack on some platforms, 0x80cdefd3 -> heap access, (the
jemalloc poison values)
- With the above look at the crash address distribution. Look at the
exception type as well. If they are not all in the same bucket you *may*
have a bad vtable or corruption occurring. Might want to do this per
platform.
- If you have a minidump see:
https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump
- If you don't have a minidump (non windows), and the line level info is
insufficient, you can grab the instruction pointer and disassemble the
binary. This will often tell you exactly what you were doing and which
value is bad but is a bit tricky to follow with the optimizer.

That's my rough workflow when looking at Soccoro. I'm leaving out the more
specific searches like driver correlation for gfx crashers for instance.

On Thu, May 26, 2016 at 9:52 AM, Benjamin Smedberg <benj...@smedbergs.us>
wrote:

[Ted Mielczarek]

On Thu, May 26, 2016, at 09:52 AM, Benjamin Smedberg wrote:
> https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/Debugging_a_minidump

I spent some time today cleaning this page up a bit. I streamlined the
Windows section (since you no longer need to manually download the
matching binaries or source), added a section about minidump-2-core for
Linux, and added a little bit of information about some useful tools I
have for additional dump analysis.

Hope that's useful to someone,
-Ted

[Ted Mielczarek]

On Thu, May 26, 2016, at 02:41 PM, Benoit Girard wrote:
> - Crash address can give you a hint as to what's going on: 0x0 -> null
> crash, 0x8 -> null offset accessing a member like this->mFoo,

N.B.: due to the way addressing works on x86-64, if the crash address is
"0x0" for a Linux/OS X crash report, or "0xffffffffffffffff" for a
Windows crash report, it's highly likely that the value is incorrect[1].
This is due to how these faults are reported from the CPU. (I have an
in-progress Breakpad patch[2] to detect this and attempt to find the
actual faulting address, but it needs some more work to land upstream.
The commit message also describes the root cause more thoroughly, if
you're interested.)

You can sanity check these crashes by clicking the "Raw Dump" tab. The
top frame of the crashing stack in that JSON data will contain the
registers from the crash context. If you see a suspicious value in one
of them (like a poison pattern) more thorough analysis may be warranted.

-Ted

1. https://bugzilla.mozilla.org/show_bug.cgi?id=974420
2.
https://github.com/luser/breakpad/commit/c8c0bbcdba178b74db171c21d2c6f4a59efd0131

[Nicholas Nethercote]

On Wed, May 25, 2016 at 4:06 PM, Nicholas Nethercote
<n.neth...@gmail.com> wrote:
>

> Do we have documentation on how to read crash reports?

I've finished a first pass at documentation on understanding
individual crash reports. The bulk of the effort was in writing this
new page: https://developer.mozilla.org/en-US/docs/Understanding_crash_reports

There are numerous "XXX" comments for things I was uncertain about. If
you know anything about crash reports, please read through and address
any of these comments you know about, or add/fix anything else as you
see fit. If you don't know anything about crash reports, please read
through and let me know if anything is unclear or could otherwise be
improved.

I also tweaked a few other pages that talk about crash reports. Full
details of what I did are here:
https://bugzilla.mozilla.org/show_bug.cgi?id=1288309#c1

I have also opened
https://bugzilla.mozilla.org/show_bug.cgi?id=1288310 about improving
documentation relating to larger-scale analysis of clusters of crash
reports. I haven't done any work on this yet.

Thanks.

Nick