Excluding memory dumps from crashpad mini dumps

[Jonathan Jones]

Hello,

I've been trying to figure out how to surgically modify the crashpad source code in order to exclude memory dumps from the minidump file entirely.

We are aware of sanitization, and we are aware this currently only works for Android.  However, when I tried to use sanitization, the minidump file sizes were unchanged ... you just ended up with much of the dump painted with "0x0defaced".  We have a strong desire to shrink the size of the minidump file.

Any help is appreciated!

Sincerely,

Jonathan Jones

[Joshua Peraza]

What information would you like to report?

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/deffd30a-a599-4c03-8a34-a4204b9baa7an%40chromium.org.

[Mark Mentovai]

If you turn on gzip compression of uploads, all of those defaced words should compress tightly.

--

[Jonathan Jones]

Well, we need to support, at a minimum, crashes on Windows (Intel), Linux (Intel), and Mac (Intel, Apple Silicon).  If we were to use sanitization, it would need to:

1. Work on all those platforms.
2. Work with a shared crash handler process _or_ a crash handler process started at client process start (and not started at process crash).

My understanding is that sanitization currently only works on Android (am I mistaken?), so we are looking for something quick we can change to get the memory out of the dumps entirely.

We _do_ need stack traces, register dumps, module lists, and annotations.

[Joshua Peraza]

You can't remove memory from the dumps entirely because that's how stack traces are generated. Crashpad doesn't report text stack traces. It reports stack memory, which is processed with debug information to generate stack traces.

Here's a blurb on hooking up sanitization for other platforms: https://groups.google.com/a/chromium.org/g/crashpad-dev/c/nWzIyKdu_4c/m/KelMda1CAwAJ

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/ec9cf2d3-35c5-42ad-a34a-7495480bb775n%40chromium.org.

[Jonathan Jones]

Is there memory besides stack-related memory being collected?  My experiments seem to indicated that we only collect memory for stacks (for example, in a 64-thread minidump, there were exactly 64 memory regions, one per thread).

[Joshua Peraza]

It is possible to configure crashpad to collect more memory, but it does not by default.

[Jonathan Jones]

I'm finding that if I don't sanitize, I see lots of Personally Identifiable Information present in the .dmp file (such as environment variable name=value pairs).

It surprised me to see the environment variables, since I wouldn't expect them to be present in the memory regions associated with thread stacks.  Is there a way to turn off the environment variable collection?

[Joshua Peraza]

Some environment variables are implicitly captured near the base of the main thread's stack memory, the arguments to main(argc, argv, envp). 

[Jonathan Jones]

Is there any way to exclude the environment variables from the minidump besides sanitizing (which doesn't yet work for any platform we care about)?

[Joshua Peraza]

Nothing out of the box. Crashpad determines how much memory to collect for stacks in crashpad/snapshot/<platform>/process_reader_<platform>.cc if you want to tinker with it.

[Jonathan Jones]

Okay, so I short-circuited some of the configuration logic and have sanitization hooked up.  On Intel Linux, Intel Mac, and Intel Windows, stacks from sanitized minidumps look reasonable.  However on Apple Silicon Mac, many of the frame addresses have been directly overwritten with the 0xdefaced scribble pattern.

I won't pretend to understand the algorithm for what sanitization does/doesn't overwrite, so do you have suggestions on where to proceed next?  Here is the code I added to handler/mac/crash_report_exception_handler.cc

kern_return_t CrashReportExceptionHandler::CatchMachException(...)
{
    ...
    process_snapshot.SetReportID(new_report->ReportID());
    // unchanged from here up

    ProcessSnapshotSanitized sanitized_snapshot;
    if (!sanitized_snapshot.Initialize(&process_snapshot,
                                       nullptr, // all annotations are allowed
                                       nullptr, // no memory ranges are allowed
                                       0,       // there is no target module
                                       true     // sanitize stacks for PII
                                       )) {
        Metrics::ExceptionCaptureResult(Metrics::CaptureResult::kSkippedDueToSanitization);
        return KERN_FAILURE;
    }

    ProcessSnapshot* snapshot_to_write = implicit_cast<ProcessSnapshot*>(&sanitized_snapshot);

    MinidumpFileWriter minidump;
    minidump.InitializeFromSnapshot(snapshot_to_write);
    AddUserExtensionStreams(user_stream_data_sources_, snapshot_to_write, &minidump);

    // unchanged from here down
    if (!minidump.WriteEverything(new_report->Writer())) {
        ...
    }
}

[Joshua Peraza]

This sounds probably related to pointer authentication.

Crashpad's stack sanitizer will destroy any data that doesn't look like a pointer pointing into one of the stack regions or loaded code. However, ARM64 pointers may have non-addressing data set in the upper bits for either memory tagging or pointer-authentication purposes and those non-address bits need to be removed from any data before considering whether they are an allowed pointer here.

The difficulty then is determining how many bits are used for addressing. I'm not sure where exactly Crashpad for ARM64 macOS stands with pointer authentication (There's https://chromium-review.googlesource.com/c/crashpad/crashpad/+/2773358). In that CL, the addressing bits are retrieved from sysctlbyname("macdep.virtual_address_size")

[Mark Mentovai]

It doesn't really stand anywhere with pointer authentication on mac-arm64, because Apple hasn't made pointer authentication (the "arm64e" ABI) available to anyone but themselves yet (without a special boot argument). But this also means that Crashpad won't see any signed pointers in your own processes.