New crash annotations API available making them almost zero-cost

[Gabriele Svelto]

TL;DR crash annotations can be recorded so that their value is read and
serialized only at crash-time, this means that even frequently updated
ones have no performance cost at runtime.

Longer version:

Crash annotations have been historically recorded in C++ using the
CrashReporter::AnnotateCrashReport() method. This would serialize the
annotation value into a string and then store it in a table that would
later be attached to a crash. A new way of recording crash annotations
is now available via CrashReporter::RegisterAnnotation<TypeName>(). This
method takes an address to the variable holding the annotation's value
(which can be a boolean, integer, string, etc...) and records it among
the annotations. At crash time we'll extract the values of the
annotations from these addresses and serialize them into the crash
report. This means that in the meantime you can update the value of the
annotation at will without having to re-record it every time. The last
value in memory will be picked up automatically if we crash.

One advantage of this system is that you can now pick a variable that is
updated very frequently, even thousands of times per second, and still
be able to record it in a crash with no performance impact. You only
have to ensure that the memory holding the value is alive when we crash,
even though if it isn't the annotation will just be ignored.

You can still record annotations the old way by using
CrashReporter::RecordAnnotation<TypeName>(). This will make a copy of
the value you passed in and keep it around with the other annotations.

You can find more details in bug 1831092 [1].

Gabriele

[1] Record all annotations using the new pull-based system and remove
the annotation table
https://bugzilla.mozilla.org/show_bug.cgi?id=1831092

[Marco Bonardo]

This is great news Gabriele.

If I'd like to register an annotation only when we crash in a specific scope (my use case is recording if a sanitized SQL query was running at crash time), do you suggest I just set and unset the registered variable, or is there also a way to unregister a variable?

Thank you,

Marco

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/83d422f9-369a-4233-a2b6-0f26c05e5a02%40mozilla.com.

[Gabriele Svelto]

On 04/03/24 14:07, Marco Bonardo wrote:
> This is great news Gabriele.
>
> If I'd like to register an annotation only when we crash in a specific
> scope (my use case is recording if a sanitized SQL query was running at
> crash time), do you suggest I just set and unset the registered
> variable, or is there also a way to unregister a variable?

You can always unregister an annotation, irrespective of if you've used
CrashReporter::RecordAnnotation<Type>() or RegisterAnnotation<Type>().
E.g. you could have a code block like:

{
uint32_t stuff_i_want_to_record = 0;
CrashReporter::RegisterAnnotationU32(Annotation::Stuff,
&stuff_i_want_to_record);
...
CrashReporter::UnregisterAnnotation(Annotation::Stuff);
}

This will incur in the cost of register/unregistering the annotation
every time you enter the scope. Alternatively you could use the
`skip_if` feature in the crash annotations list. If you know that your
annotation should be disregarded if it has a certain value you can put
it in the annotations list and it will be automatically skipped when it
matches that value.

See this as an example:

https://searchfox.org/mozilla-central/rev/202c48686136360a23b73a49b611a19e64f3e1b8/toolkit/crashreporter/CrashAnnotations.yaml#883

We always record the amount of textures we're using, but if the value is
zero the annotation gets skipped.

Last but not least, we'll auto-skip the annotation for string pointer
types if the pointed-to strings are empty or NULL (though I might not
have surfaced all the string-related methods in C++ yet; the
functionality is in a Rust crate that's wrapped in the crash reporter code).

Gabriele

[Daniel Holbert]

Thanks Gabriele -- this seems great for making crash reports more useful!

Can you provide guidance (or link to documentation) for how to locally test a newly added annotation, to be sure it's working properly?  i.e. once I've added an annotation, how might I go about generating a dummy crash report so that I can make sure my annotation shows up?

Presumably I might add a `MOZ_CRASH("crashing");` to the relevant section of code, and then I'd run my local build and crash, or something along those lines -- but right now, my local build never gives me a reporter dialog, including for a local opt build with `--enable-crashreporter`, so I'm not entirely sure how to generate a crash report.

Is there another mozconfig flag I should be using?  (and, stepping back: is it OK to submit crash reports from local builds for stuff like this, or is there a preferred alternative approach you'd rather that folks use, if this would be viewed as pollution of crash-stats?)

Thanks again,

~Daniel

--
You received this message because you are subscribed to the Google Groups "dev-pl...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-platform...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/5c7557db-65ec-4b4c-ac54-a72ada176ce4%40mozilla.com.

[Nick Alexander]

Hi Daniel, others,

On Tue, Mar 5, 2024 at 4:14 PM Daniel Holbert <dhol...@mozilla.com> wrote:

Thanks Gabriele -- this seems great for making crash reports more useful!

Can you provide guidance (or link to documentation) for how to locally test a newly added annotation, to be sure it's working properly?  i.e. once I've added an annotation, how might I go about generating a dummy crash report so that I can make sure my annotation shows up?

It's not particularly easy, but it's also not particularly hard, to verify that crashes are collected (for example, https://bugzilla.mozilla.org/show_bug.cgi?id=1679440) and that annotations are correct (e.g., https://bugzilla.mozilla.org/show_bug.cgi?id=1697875).  I do recall that it was not worth the hassle to locate the tests next to the tested feature; it was much easier to put them next to the existing crash tests.

I will also caution that even once the annotations are in place, it requires additional work to make them useful: e.g., something like https://bugzilla.mozilla.org/show_bug.cgi?id=1796132 is required.

Hopefully others will tell me that this dance is simpler now :)

Best,

Nick

[Daniel Holbert]

Thanks! To clarify, I'm not talking about writing automated tests - just how I might manually spot-testing that an intentional crash in the appropriate spot (in my own local build) does indeed include annotations that I added nearby.

Right now I'm stuck at the "how do I get the crash reporter to show up" phase, which is maybe easy to address, but I'm also curious if this is the right sort of thing to do in the first place vs. if there are easier/recommended ways to preview the annotations that have been recorded.

[Gabriele Svelto]

Hi Daniel,
I'm sorry that our crash reporting documentation is not really
up-to-date. We've been doing some very large refactoring and the docs
didn't catch up, it's one of my goals to improve them this year.

In the meantime the easiest way to check if your crash annotation is
present is the following:

- Add the annotation to `toolkit/crashreporter/CrashAnnotations.yaml`
- Build with `ac_add_options --enable-crashreporter` in your .mozconfig
(which you already did)
- Run with `./mach run --enable-crash-reporter` (notice the different
dash compared to the mozconfig, sorry about that, historic artifact)
- Crash the process you care about: you can use about:crashparent for
the parent process, about:crashcontent for content processes, ... If the
process doesn't have code dedicated to make it crash you can always
crash it via `kill -ABRT <pid>` on Linux and macOS. This will trigger
the crash reporter
- At this stage you'll be able to submit the crash but I don't recommend
it as Socorro won't show you new annotations by default unless you have
protected data access. Instead, before interacting with the crash
reporter dialog look for the crash report files. They'll be under
`~/.mozilla/firefox/Crash Reports/pending` on Linux,
`%UserProfile%/AppData/Roaming/Mozilla/Firefox/Crash Reports/pending/`
on Windows and `~/Library/Application Support/Mozilla/Firefox/Crash
Reports/pending` on macOS). Each crash report is made of two files
<uuid>.dmp and <uuid>.extra. The .extra file is JSON and will contain
the annotations.

If for some reason you still need to submit the crashes then it's a
better idea to use Socorro staging environment. Go to about:config and
change the `breakpad.reportURL` pref from
`https://crash-stats.mozilla.org/report/index/` to
`https://crash-stats.allizom.org/report/index/`.

Gabriele

On 06/03/24 04:52, Daniel Holbert wrote:
> Thanks! To clarify, I'm not talking about writing automated tests - just
> how I might manually spot-testing that an intentional crash in the
> appropriate spot (in my own local build) does indeed include annotations
> that I added nearby.
>
> Right now I'm stuck at the "how do I get the crash reporter to show up"
> phase, which is maybe easy to address, but I'm also curious if this is
> the right sort of thing to do in the first place vs. if there are
> easier/recommended ways to preview the annotations that have been recorded.
>
> On Tue, Mar 5, 2024, 7:09 PM Nick Alexander <nalex...@mozilla.com
> <mailto:nalex...@mozilla.com>> wrote:
>
> Hi Daniel, others,
>
> On Tue, Mar 5, 2024 at 4:14 PM Daniel Holbert <dhol...@mozilla.com

> <mailto:dhol...@mozilla.com>> wrote:
>
> Thanks Gabriele-- this seems great for making crash reports more

> useful!
>
> Can you provide guidance (or link to documentation) for how to
> locally test a newly added annotation, to be sure it's working
> properly?  i.e. once I've added an annotation, how might I go
> about generating a dummy crash report so that I can make sure my
> annotation shows up?
>
>
> It's not particularly easy, but it's also not particularly hard, to
> verify that crashes are collected (for example,
> https://bugzilla.mozilla.org/show_bug.cgi?id=1679440

> <https://bugzilla.mozilla.org/show_bug.cgi?id=1679440>) and that

> annotations are correct (e.g.,
> https://bugzilla.mozilla.org/show_bug.cgi?id=1697875

> <https://bugzilla.mozilla.org/show_bug.cgi?id=1697875>).  I do