Exposing a MindumpWriteDump-equivalent?

[Sigurður Ásgeirsson]

Hey y'all,

to try and figure out what's going on with https://crbug.com/806661, I implemented a quick hack to get browser-triggered dumps of renderers under very specific conditions. See <https://chromium-review.googlesource.com/c/chromium/src/+/950127> - the idea here is to try and act quickly to grab a dump, as we believe the problem is a manifestation of a shutdown race, or a late-stage shutdown hang. To that end I've re-purposed the old hack I wrote to massage MDWD-generated dumps into a (barely) Crashpad-acceptable shape.

I think this'd be a very handy primitive to have in Chrome, as often we've wished to be able to know WTH is going on in e.g. the GPU process or a renderer when specific conditions arise in the browser.

It then occurred to me that there's no reason not to use Crasphad's snapshotting, but many good reasons to do so, as that way we'll get all the Crashpad metadata out of the process, and it'd be easy to reuse crumbs scraping and such.

So, I wonder, how about exposing a MinidumpWriteDump-equivalent function from Crashpad? I figure this'd provide the same extensibility offered in the handler main, would take care of dropping the dump into an existing Crashpad database. It could then also record consistent metrics, etc.

NB: the downside to dropping dumps into the Crashpad database from "the outside" like this is that the upload will be delayed. I wonder if there's an easy way to fix that, by e.g. having the uploader watch for changes in the DB directory?

Siggi

[Robert Sesek]

Hey Siggi,

Could you instead use the crashpad::CrashReportDatabase API directly? PrepareNewCrashReport() gives you an object with an open file handle, to which you can write the MDWD contents, and then FinishedWritingCrashReport() will stage it for uploading. Dropping files directly into the Crashpad database should be avoided, and I do not think is something we will support.

I'll leave it to Mark to comment on a MinidumpWriteDump-equivalent.

- Robert

rsesek / @chromium.org

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev+unsubscribe@chromium.org.
To post to this group, send email to crashp...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAJfD1ZHoVbKYYruzwG3SRQ3E44Nj8Bs2aoApnR1SphP8EnDHGg%40mail.gmail.com.

[Sigurður Ásgeirsson]

On Thu, Mar 8, 2018 at 12:05 PM Robert Sesek <rse...@chromium.org> wrote:

Hey Siggi,

Could you instead use the crashpad::CrashReportDatabase API directly? PrepareNewCrashReport() gives you an object with an open file handle, to which you can write the MDWD contents, and then FinishedWritingCrashReport() will stage it for uploading.

This is what my hack does, although the MDWD-produced minidump has to be massaged a bit to endow it with the minimal in-band metadata Crashpad required.

[Mark Mentovai]

This is close to crashpad::CrashpadClient::DumpAndCrashTargetProcess(), which is Windows-only for architectural reasons. It sounds like you’re asking for a variant without the “and crash”.

We can certainly do that. DumpAndCrashTargetProcess() does some gnarly thread-injection RaiseException()-calling stuff to achieve its ends. There are a few reasons for this, more details are available in the history on the initial review.

Following that template, we could do a DumpWithoutCrash()-calling variant in place of the RaiseException()-calling one. But for the non-crash case, we may not need to get that gnarly: we could instantiate a snapshot from a HANDLE anywhere, and have the minidump be written into a database from any process. On the Chrome side, we’d still need to do a bit of a dance to make sure that the client ID and the “process annotations,” the --annotation parameters that we start crashpad_handler with, properly make their way into the dump. The whole procedure would essentially be equivalent to crashpad::CrashReportExceptionHandler::ExceptionHandlerServerException(), which shows where these are pushed into the process snapshot. It’s fairly straightforward, but it does turn out to be a little more work than “just” injecting a thread, it won’t have the same sorts of extensibility hook setup as the real crashpad_handler (unless still more work is undertaken), it’s possible that said extensibility may not even be appropriate to run from within a Chrome browser, and it still has the delayed-upload problem you mention.

The plan for upload is to listen to the filesystem to provide better information on when to look for new work. This is bug 128. We do intend for multiple database writers to “feed” a single uploader, so it’s desirable to get rid of that 15-minute (maximum) delay.

--

You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.

To post to this group, send email to crashp...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAJfD1ZHWo_gA1SV9%3D1qyn7N_jyJaJp8fbKJ2F0J%3DDnBExiw8pw%40mail.gmail.com.

[Sigurður Ásgeirsson]

On Fri, Mar 9, 2018 at 5:19 PM Mark Mentovai <ma...@chromium.org> wrote:

This is close to crashpad::CrashpadClient::DumpAndCrashTargetProcess(), which is Windows-only for architectural reasons. It sounds like you’re asking for a variant without the “and crash”.

Well, it's a little more intricate - see below...

 

We can certainly do that. DumpAndCrashTargetProcess() does some gnarly thread-injection RaiseException()-calling stuff to achieve its ends. There are a few reasons for this, more details are available in the history on the initial review.

Yeah, the main objections to using this for this case is that:

• It relies on a whole lot of machinery in the process to be initialized and whole to some extent, including e.g. the crashpad handler and exception dispatching.
• We're (pretty sure that we're) racing the process' exit here, so the quicker we can suspend the process the better.
  The inject-and-crash method needs to spin up a thread in the target, trigger an exception and wait for the handler to service it, all of which reduces the odds of capture.
• It's not possible to adorn (or override) the crash metadata with additional info specific to the crash.

 

Following that template, we could do a DumpWithoutCrash()-calling variant in place of the RaiseException()-calling one. But for the non-crash case, we may not need to get that gnarly: we could instantiate a snapshot from a HANDLE anywhere, and have the minidump be written into a database from any process. On the Chrome side, we’d still need to do a bit of a dance to make sure that the client ID and the “process annotations,” the --annotation parameters that we start crashpad_handler with, properly make their way into the dump. The whole procedure would essentially be equivalent to crashpad::CrashReportExceptionHandler::ExceptionHandlerServerException(), which shows where these are pushed into the process snapshot. It’s fairly straightforward, but it does turn out to be a little more work than “just” injecting a thread, it won’t have the same sorts of extensibility hook setup as the real crashpad_handler (unless still more work is undertaken), it’s possible that said extensibility may not even be appropriate to run from within a Chrome browser, and it still has the delayed-upload problem you mention.

Yups, but it does at least allow for the possibility of these things, and all the pieces exist :).

The gnarliest bit, though IMHO, is to make sure that we never grab crashes for non-Chrome processes. On Windows that's "pretty easy", as if you have a handle to a process, its PID won't be reused, and so the identity of our own sub processes is relatively easily validated and preserved. Maybe this could be further ensured by verifying the presence of at least one Crashpad ClientInfo in the dumped process?

The plan for upload is to listen to the filesystem to provide better information on when to look for new work. This is bug 128. We do intend for multiple database writers to “feed” a single uploader, so it’s desirable to get rid of that 15-minute (maximum) delay.

On Thu, Mar 8, 2018 at 12:48 PM Sigurður Ásgeirsson <si...@chromium.org> wrote:

On Thu, Mar 8, 2018 at 12:05 PM Robert Sesek <rse...@chromium.org> wrote:

Hey Siggi,

Could you instead use the crashpad::CrashReportDatabase API directly? PrepareNewCrashReport() gives you an object with an open file handle, to which you can write the MDWD contents, and then FinishedWritingCrashReport() will stage it for uploading.

This is what my hack does, although the MDWD-produced minidump has to be massaged a bit to endow it with the minimal in-band metadata Crashpad required.

--

You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To post to this group, send email to crashp...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAJfD1ZHWo_gA1SV9%3D1qyn7N_jyJaJp8fbKJ2F0J%3DDnBExiw8pw%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To post to this group, send email to crashp...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAHk1GDzvab%2BerDzWBi-%3DX7Uy3d9Ax_UM9ooBOHp%3D%2BQyjGu2ZEA%40mail.gmail.com.