General questions about crashpad_handler behavior

[Matthew Spelman]

I'm working on integrating Crashpad into a product environment that is quite complex, including multiple application processes, etc. and I have a few questions about how the handler behaves. Based on the documentation, it looks like the handler normally gets kicked off in its own process, subject to platform limitations. That being said:

1. Does the handler continue running after the application which started the handler exits? The documentation for StartHandler() doesn't really specify this, though the doc for StartHandlerAtCrash() seems to indicate that the handler exits after generating a report if I start the handler by calling that function, though that function only seems to be available on Linux. My application runs on Linux (both Intel and ARM), Mac (both Intel and ARM), and Windows, so the ins and outs of the behavior on those platforms is a little murky to me.

2. What happens if I try to call StartHandler() or StartHandlerAtCrash() when a handler is already running? Does the call fail? Does it start another handler? Does it re-use the existing handler?

3. If I have multiple copies of my application running and they're connected to the same handler, if they both crash, are two reports generated? Or does the second report get lost?

4. When my application exits do I need to perform some kind of shutdown/cleanup for the handler?

5. What are the performance costs of adding the initialization of a Crashpad handler? I see that I can start a handler asynchronously on Windows, but what about the other platforms? Is there a performance benefit to using StartHandlerAtCrash() on Linux compared to StartHandler()?

Sorry if these are documented somewhere, but I've not been able to find them.

Thanks,

Matt

[Joshua Peraza]

On Fri, Aug 19, 2022 at 1:46 PM Matthew Spelman <mjs...@gmail.com> wrote:

I'm working on integrating Crashpad into a product environment that is quite complex, including multiple application processes, etc. and I have a few questions about how the handler behaves. Based on the documentation, it looks like the handler normally gets kicked off in its own process, subject to platform limitations. That being said:

1. Does the handler continue running after the application which started the handler exits? The documentation for StartHandler() doesn't really specify this, though the doc for StartHandlerAtCrash() seems to indicate that the handler exits after generating a report if I start the handler by calling that function, though that function only seems to be available on Linux. My application runs on Linux (both Intel and ARM), Mac (both Intel and ARM), and Windows, so the ins and outs of the behavior on those platforms is a little murky to me.

Generally speaking the handler process exits when all of its connected clients have exited. If the process that started crashpad_handler has exited but it has launched children, which inherited a connection to crashpad_handler, crashpad will continue to monitor those children. StartHandlerAtCrash() is special in that it passes the flag --trace-parent-with-exception to crashpad_handler (docs). crashpad_handler will exit after dumping the parent process and possibly uploading a report because there was no socket connection created between the parent and crashpad_handler and so no way for the parent to ever connect more children to this instance of crashpad_handler.

 

2. What happens if I try to call StartHandler() or StartHandlerAtCrash() when a handler is already running? Does the call fail? Does it start another handler? Does it re-use the existing handler?

For Linux:

The StartHandler() functions can be called multiple times in the life of a process but only once by each loaded code module. e.g. if you link crashpad's :client  library into your application's libfoo.so, it may call any one of the StartHandler() methods, but not multiple. Each method will record some global state in libfoo.so and calling one of the other methods will overwrite that state, but should hit this dcheck. If you ship a second library or some other piece of software you integrate with ships a library libbar.so which also links crashpad/client, it may also call one of the StartHandler() methods and it will save its state in libbar.so. One of the pieces of state that crashpad saves when it installs a signal handler is what is the previous signal handler that was installed with the OS. After any one of the crashpad signal handlers handle a crash signal, they will restore the previously installed signal handler and re-raise the signal, giving all the handlers a chance to run. Each of the handler instances would be different. You can have a crashpad client re-use a crashpad_handler by calling CrashpadClient::SetHandlerSocket().

For Windows, there's no such chaining of exception handlers (I'm pretty sure) so a second module calling StartHandler() or directly calling SetUnhandledExceptionFilter() would break the first crashpad client. I'm not sure what would happen on macOS.

3. If I have multiple copies of my application running and they're connected to the same handler, if they both crash, are two reports generated? Or does the second report get lost?

You should get both reports.

 

4. When my application exits do I need to perform some kind of shutdown/cleanup for the handler?

No.

 

5. What are the performance costs of adding the initialization of a Crashpad handler? I see that I can start a handler asynchronously on Windows, but what about the other platforms? Is there a performance benefit to using StartHandlerAtCrash() on Linux compared to StartHandler()?

I've not personally felt crashpad_handler startup time to be a performance issue. StartHandlerAtCrash() allows more complexity at crash time to avoid the memory cost of a long-lived crashpad_handler or risk crashpad_handler being terminated before it can handle any real crashes in low-memory conditions.

 

Sorry if these are documented somewhere, but I've not been able to find them.

Thanks,

Matt

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/3191eb4d-c07b-4705-b6c7-3ccf9d75cc0an%40chromium.org.

[Joshua Peraza]

On Fri, Aug 19, 2022 at 3:00 PM Joshua Peraza <jpe...@google.com> wrote:

On Fri, Aug 19, 2022 at 1:46 PM Matthew Spelman <mjs...@gmail.com> wrote:

I'm working on integrating Crashpad into a product environment that is quite complex, including multiple application processes, etc. and I have a few questions about how the handler behaves. Based on the documentation, it looks like the handler normally gets kicked off in its own process, subject to platform limitations. That being said:

1. Does the handler continue running after the application which started the handler exits? The documentation for StartHandler() doesn't really specify this, though the doc for StartHandlerAtCrash() seems to indicate that the handler exits after generating a report if I start the handler by calling that function, though that function only seems to be available on Linux. My application runs on Linux (both Intel and ARM), Mac (both Intel and ARM), and Windows, so the ins and outs of the behavior on those platforms is a little murky to me.

Generally speaking the handler process exits when all of its connected clients have exited. If the process that started crashpad_handler has exited but it has launched children, which inherited a connection to crashpad_handler, crashpad will continue to monitor those children. StartHandlerAtCrash() is special in that it passes the flag --trace-parent-with-exception to crashpad_handler (docs). crashpad_handler will exit after dumping the parent process and possibly uploading a report because there was no socket connection created between the parent and crashpad_handler and so no way for the parent to ever connect more children to this instance of crashpad_handler.

 

2. What happens if I try to call StartHandler() or StartHandlerAtCrash() when a handler is already running? Does the call fail? Does it start another handler? Does it re-use the existing handler?

For Linux:

The StartHandler() functions can be called multiple times in the life of a process but only once by each loaded code module. e.g. if you link crashpad's :client  library into your application's libfoo.so, it may call any one of the StartHandler() methods, but not multiple. Each method will record some global state in libfoo.so and calling one of the other methods will overwrite that state, but should hit this dcheck. If you ship a second library or some other piece of software you integrate with ships a library libbar.so which also links crashpad/client, it may also call one of the StartHandler() methods and it will save its state in libbar.so. One of the pieces of state that crashpad saves when it installs a signal handler is what is the previous signal handler that was installed with the OS. After any one of the crashpad signal handlers handle a crash signal, they will restore the previously installed signal handler and re-raise the signal, giving all the handlers a chance to run. Each of the handler instances would be different. You can have a crashpad client re-use a crashpad_handler by calling CrashpadClient::SetHandlerSocket().

Also, this chaining depends on any other signal using software in the process co-operating in restoring and re-raising signals. If you're on Android, libsigchain might help mediate this.

[Matthew Spelman]

Thanks for the info!

[Matthew Spelman]

Following up on this: Is there a way to forcibly terminate a crashpad_handler from the handler API? I'm trying to measure the performance of the handler startup using Google benchmark, which simply kicks off the handler with some default arguments. If I can ensure that the handler is terminated on each pass through the loop, that would verify I'm actually testing what I need to.

Thanks.