Add annotations to crash dump after crashpad_client.StartHandler()

380 views
Skip to first unread message

Mayur Kale

unread,
Jan 12, 2021, 7:13:54 AM1/12/21
to Crashpad-dev

Hi guys,

We have integrated a crash pad to our product to collects crash dumps from client-side.

One way of adding annotations to crash dumps is to add them at the time of starting crash_handler in method “crashpad_client.StartHandler(..,.., annotations,…)”.

I want to know whether we can add new annotations after the crashpad handler started and how?. It would be great if you could share an example code snippet. 

I highly appreciate your reply.

-Mayur

Joshua Peraza

unread,
Jan 12, 2021, 5:11:37 PM1/12/21
to Mayur Kale, Crashpad-dev
The test code here demonstrates how you can set annotations from the client either before or after starting the handler.

Lines 314-315 are before starting the handler, but can be moved below the call to InstallHandler() and still function correctly.

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/390c7f5a-8cb7-4e5a-8319-4b4f39009352n%40chromium.org.

Mayur Kale

unread,
Jan 13, 2021, 12:17:14 AM1/13/21
to Crashpad-dev, Joshua Peraza, Crashpad-dev, Mayur Kale
Hi Joshua,

Thank you so much for your reply.
I am trying you suggested code as follows :

bool startHandler(CrashpadClient& client)
{
    std::map<std::string, std::string> annotations;
    annotations["product version"] = "V1";

    std::vector<std::string> arguments;

    base::FilePath db("/home/mayur/CrashPad-data/crashpad_db");
    base::FilePath handler("/home/mayur/crashpad/crashpad_handler");
    std::unique_ptr<CrashReportDatabase> database = CrashReportDatabase::Initialize(db);
    database->GetSettings()->SetUploadsEnabled(false);
    std::string url;
    return client.StartHandler(handler,
            db,
            db,
            url,
            annotations,
            arguments,
            false,
            false);
}

void crash()
{
    int *a= NULL;
    *a=3;
}

void initiate_crash()
{
    cout << "Initiating Crash" <<endl;
    crash();
}

int main()
{
  constexpr char kTestAnnotationName[] = "name_of_annotation";
  constexpr char kTestAnnotationValue[] = "value_of_annotation";
  constexpr char kTestAnnotationName1[] = "name_of_annotation1";
  constexpr char kTestAnnotationValue1[] = "value_of_annotation1";

    cout << "starting crashpad handler" << endl;

    CrashpadClient Initial_client;
    //int handler_sock, handler_pid;
    
    crashpad::AnnotationList::Register();

    static StringAnnotation<32> test_annotation(kTestAnnotationName);
    test_annotation.Set(kTestAnnotationValue);
    
    startHandler(Initial_client);

    static StringAnnotation<32> test_annotation1(kTestAnnotationName1);
    test_annotation1.Set(kTestAnnotationValue1);

    //Initial_client.GetHandlerSocket(&handler_sock, &handler_pid);
    
    initiate_crash();
    return 0;
}

My output is as follows : 

MDRawCrashpadInfo
  version = 1
  report_id = ed05206a-261a-40fa-ad28-2b6971a4463f
  client_id = e6d0734c-70b0-46f7-a033-8a8a73c96239
  simple_annotations["product version"] = V1
  module_list[0].minidump_module_list_index = 0
  module_list[0].version = 1

I didn't see my annotations in the dump.
I am using "minidum_dump dump_file " to see annotations.

Am I missing something here?

-Mayur

Joshua Peraza

unread,
Jan 13, 2021, 12:46:58 AM1/13/21
to Mayur Kale, Crashpad-dev

Mayur Kale

unread,
Jan 13, 2021, 1:28:32 AM1/13/21
to Joshua Peraza, Crashpad-dev
minidump_dump does not show all module annotations in the dump:
I thought that they have extended minidump_dump code to show these module annotations as well. Please correct me if I am wrong.
>> Could you please help me with how to use this code to see my annotations?                   

-Mayur
--
Thanks
Mayur

“Everybody in this country should learn how to program a computer… because it teaches you how to think”
—Steve Jobs

Fabio Corrirossi

unread,
Jan 25, 2021, 9:46:30 AM1/25/21
to Crashpad-dev, mayur...@gmail.com, Crashpad-dev, Joshua Peraza
Hello everyone,

do you have news about this?
I am trying to achieve the same (pushing annotations after starting the handler) but no luck so far.

I am also trying something like this:

crashpad::AnnotationList::Register();
//...
crashpad::AnnotationList* list = crashpad::AnnotationList::Get();
static std::string test= "test_value";
static crashpad::Annotation annotation(crashpad::Annotation::Type::kString, "test_key", & test  );
list->Add(&annotation);

But I can't see any annotations coming.
I am using crashpad connected to backtace at the moment and if I put annotations in the StartHandler I can see them ok in the crash report.

Many thanks in advance,
Fabio

Mayur Kale

unread,
Jan 27, 2021, 11:20:36 AM1/27/21
to Crashpad-dev, Fabio Corrirossi, Mayur Kale, Crashpad-dev, Joshua Peraza
Hi,

Following code worked for me.

void microsoft::crash_handler::add_crash_annotations(const std::map<stdext::tstring, stdext::tstring>& annotations) {
    CrashpadInfo* crashpad_info = CrashpadInfo::GetCrashpadInfo();
    SimpleStringDictionary* module_annotations =
        crashpad_info->simple_annotations();
    if (!module_annotations) {
      module_annotations = new SimpleStringDictionary();
      crashpad_info->set_simple_annotations(module_annotations);
    }

    for (const auto& iterator : annotations) {
      module_annotations->SetKeyValue(iterator.first.c_str(),
                                      iterator.second.c_str());
    }
}

-Mayur

Matthew Spelman

unread,
Feb 15, 2023, 11:14:35 AM2/15/23
to Crashpad-dev, Mayur Kale, Fabio Corrirossi, Crashpad-dev, Joshua Peraza
I seem to be having an issue with module-level annotations appearing in minidumps on Linux. Process-level annotations work fine, however. For example, the minidump_dump output looks like this:

MDRawCrashpadInfo
  version = 1
  report_id = e2832f67-d146-443a-b449-5e6ca200e1b6
  client_id = 443e2cd9-eb71-44fa-9249-fa0df609ca8e
  simple_annotations["format"] = minidump
  module_list[0].minidump_module_list_index = 803


  module_list[0].version = 1

  address_mask = 0


"format" is a process-level annotation, but there's a whole set of module-level annotations being set via a simplestringdictionary (using the code from Mayur's previous post as a guide) I am not seeing.

product.version.number
product.root

etc, which I have searched the entire minidump output for. I saw there was an issue with minidump_dump showing module-level annotations (https://bugs.chromium.org/p/crashpad/issues/detail?id=329&q=minidump_dump&can=2), but we're seeing them using minidump_dump on Mac and Windows, so I'm not sure if that's still an open issue.

Could it be possible that PII sanitization is scrubbing the module-level annotations on Linux? Or is there a known issue with module-level annotations on Linux? I tried searching the forum here but this is the only thread that seemed relevant.

Thanks,
Matt

Joshua Peraza

unread,
Feb 15, 2023, 3:06:08 PM2/15/23
to Matthew Spelman, Crashpad-dev, Mayur Kale, Fabio Corrirossi
It is possible for sanitization to remove annotations. The sanitizer takes a vector of allowed annotations (nullptr to allow all): https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/snapshot/sanitized/process_snapshot_sanitized.h;drc=9369044d135466db00f0abb5828f55821c0341d8;l=71

Breakpad recently received support for showing module annotations, so make sure you've got this patch: https://chromium-review.googlesource.com/c/breakpad/breakpad/+/4195756

`strings` is also usually successful at finding annotations if they are there.

Matthew Spelman

unread,
Feb 15, 2023, 5:04:55 PM2/15/23
to Crashpad-dev, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi, Matthew Spelman
Hi Joshua,

Thanks, I verified I have the updated Breakpad (not using actual Breakpad, just the minidump_dump tool to look at minidumps from Crashpad), and I've also tried looking at the minidumps with the strings command, but I'm still not seeing the module-level annotations I am setting via the simplestringdictionary. I also tried adding an annotation via the .set method as demonstrated in https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/client/annotation_test.cc;l=84;drc=761ff9773d30f6f82382516b47e9f3af2a5e216c?q=annotation&ss=chromium%2Fchromium%2Fsrc:third_party%2Fcrashpad%2Fcrashpad%2Fclient%2F but I'm still not seeing it in the minidumps on Linux. I've also verified that sanitization is off. Is there any way the annotations could be showing up in Mac and Windows minidumps, but not Linux? Is there some OS-specific weirdness that can happen?

Thanks,
Matt

Joshua Peraza

unread,
Feb 16, 2023, 4:12:28 PM2/16/23
to Matthew Spelman, Crashpad-dev, Mayur Kale, Fabio Corrirossi

Matthew Spelman

unread,
Feb 16, 2023, 5:02:31 PM2/16/23
to Crashpad-dev, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi, Matthew Spelman
The test passes when I build crashpad, so I think so:
[104/488] CXX obj/client/client_test.crashpad_client_linux_test.o

I even added this code to my application, but couldn't find this annotation in the resulting minidump:

crashpad::AnnotationList::Register();
static crashpad::StringAnnotation<32> test_annotation("name_of_annotation");
test_annotation.Set("value_of_annotation");

I'm looking at both of these sections, as well as using strings [dump file] | grep "name_of_annotation" on the minidump file.

module[0]
MDRawModule
[list of data not including "name_of_annotation]

MDRawCrashpadInfo
  version = 1
  report_id = 5b631272-9a53-4f7c-9b29-2a2083b1a232
  client_id = 443e2cd9-eb71-44fa-9249-fa0df609ca8e


So I think the annotations are being set correctly, they're just not making it into the file, somehow.

Matthew Spelman

unread,
Feb 22, 2023, 12:19:31 PM2/22/23
to Crashpad-dev, Matthew Spelman, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi
Also, are there any tradeoffs to using TSimpleStringDictionary vs. AnnotationsSimpleMap vs. AnnotationsVector vs. AnnotationObjects? I've tried setting annotations with each of these methods  but none of them are making it into the minidump files on Linux.

Matthew Spelman

unread,
Feb 23, 2023, 7:29:23 PM2/23/23
to Crashpad-dev, Matthew Spelman, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi
I've found that the annotations appear on Linux if I build Crashpad in debug mode, so there must be something in release mode that's optimizing them away or something. Possibly due to changes I had to make to get Crashpad to build with gcc instead of clang. Investigation continues!

Joshua Peraza

unread,
Feb 23, 2023, 8:10:04 PM2/23/23
to Matthew Spelman, Crashpad-dev, Mayur Kale, Fabio Corrirossi
Verify crashpad's note is being linked to your application: https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/client/crashpad_info_note.S
You can view notes with `readelf --notes <binary>`. 
The note is used to find your application's CrashpadInfo, which links to annotations.

Matthew Spelman

unread,
Feb 24, 2023, 1:47:17 PM2/24/23
to Crashpad-dev, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi, Matthew Spelman
Yup, it looks like the note is not being linked in when we use a release build of Crashpad, only the debug version.

Joshua Peraza

unread,
Feb 24, 2023, 4:53:37 PM2/24/23
to Matthew Spelman, Crashpad-dev, Mayur Kale, Fabio Corrirossi
Perhaps the reference here is being optimized away: https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/client/crashpad_info.cc;drc=9369044d135466db00f0abb5828f55821c0341d8;l=105

There may be some pragmas you could use to disable certain optimizers for gcc.
Message has been deleted

Matthew Spelman

unread,
Feb 28, 2023, 10:32:56 AM2/28/23
to Crashpad-dev, Joshua Peraza, Crashpad-dev, Mayur Kale, Fabio Corrirossi, Matthew Spelman
Did a bit more digging into this and I think I found a solution. GCC is indeed optimizing away the pointer_to_note_section you mentioned in your last message. I was able to get around this by adding the "used" attribute to the reference, so that it looks like this:
 __attribute__((used)) static volatile int* pointer_to_note_section = CRASHPAD_NOTE_REFERENCE;

This might be a good change to make to the repo so that anyone else who builds Crashpad with GCC doesn't run into this issue.

Thanks for the help,
Matt
Reply all
Reply to author
Forward
0 new messages