Get crash dumps for Sandboxed processes

[Sandeep Avula]

Hi all, 

What is the standard AOSP procedure for gathering crash dumps from sandboxed processes? On our Android Pie devices which are using Android WebView, we are encountering an issue where we couldn’t collect the tombstone dump file for sandboxed processes and it results in the error message: “E libc: failed to connect to tombstoned: Operation not permitted”.

I suspect it might be due to socket system call restrictions placed in seccomp-bpf. After allowing those system calls I was able to collect the tombstone dump for the Webview sandboxed processes. I was thinking that the SandboxedHandler is broken and not being able to communicate with the Crashpad Handler since it is running in a sandboxed environment. Any suggestions or thoughts on how to obtain tombstone dump files for sandboxed processes ?

Thanks,
Sandeep

[Torne (Richard Coles)]

Hi Sandeep,

Tombstones used to work via the "fallback" dumping process (in-process, without requiring the socket connection), but that seems to have broken at some point and I'm not currently sure why unfortunately. We don't want to allow connecting to new sockets from the renderer for security reasons.

For the Google-branded WebView we don't use Android crash dumps for the renderer process at all as we rely on Crashpad instead, and we intentionally prevent the system's default crash handler from being called at all, so we unfortunately didn't notice that this was broken for a while. For AOSP builds Crashpad isn't used by default.

It's not super clear from your message whether you're expecting a Crashpad dump or an Android tombstone - you're referring to parts of both. Do you enable Crashpad in your builds?

--
You received this message because you are subscribed to the Google Groups "android-webview-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to android-webview...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/android-webview-dev/72623892-d41a-402f-baf6-5d2303662082n%40chromium.org.

[Sandeep Avula]

Hi Torne, 

Thank you for your prompt response. 

Do you enable Crashpad in your builds?

We do not enable crashpad in our AOSP builds as it follows the default build configuration

I was expecting the Android Tombstones for the renderer process. I was confused earlier thinking that Crashpad code is responsible writing the tombstones. I have the following questions.

1.If Crashpad is not enabled, then is Android responsible for capturing the renderer process crashes and writing tombstones? Is this 'fallback' currently broken and is there a bug tracking this issue?

2.  Do you have guidance on where to look or what component was previously responsible for enabling tombstones for renderer processes?

3. How to enable Crashpad by default in our AOSP builds ? Any documentation would also help ?

Thanks,
Sandeep

[Sandeep Avula]

Friendly ping 

[Nate Fischer]

https://chromium.googlesource.com/chromium/src/+/HEAD/android_webview/docs/aosp-system-integration.md is our official documentation, but I don't think we have anything which covers enabling crashpad.

Nate Fischer | Software Engineer | ntf...@google.com

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/android-webview-dev/9d85522c-7070-4dab-88bf-f9371615c910n%40chromium.org.

[Torne (Richard Coles)]

On Thu, 12 Oct 2023 at 20:01, Sandeep Avula <avulasa...@gmail.com> wrote:

Hi Torne, 

Thank you for your prompt response. 

Sorry, I didn't get back to this for a while and haven't had a chance to test things out locally to get confident answers unfortunately. 

Do you enable Crashpad in your builds?

We do not enable crashpad in our AOSP builds as it follows the default build configuration

I think crashpad might actually always be enabled in WebView at this point, even in open source builds with the default config, but haven't had a chance to verify this.

What used to happen is that we would first check whether the user had consented to metrics/diagnostic info sharing in Android before deciding whether to enable crashpad at runtime. The code to check for this consent is not in the open source repo and so in an open source build this always returned false. I don't think we actually had any *build time* switch for whether crashpad should be enabled or not, because this runtime check meant there wasn't really a pressing need.

However, when we introduced the WebView developer UI and base94-encoded minidumps in logcat, we changed the behavior here: we wanted to always generate a dump and send it to logcat and to the WebView service, and then the service would check the consent to decide whether to actually upload the dump or not. This meant that we could still get the dump from logs that were shared with us directly, or that were uploaded manually through the developer UI, even if the automatic uploads were disabled.

So I think what *might* now be happening for open source builds is the same: the crashpad handler is probably always installed and runs to generate a dump to logcat/the service, and it's just that the dumps are never then actually *uploaded* anywhere. If this is the case then we should probably fix this, because this is not very helpful for anyone else building WebView - creating crashpad dumps and then not uploading them anywhere is worse than just letting android's crash handler run.

In any case: what do you actually get in the logs when a crash happens? Can you share a full example log for the kind of crash you're concerned with? That would make it easier to figure out what's going on.

I was expecting the Android Tombstones for the renderer process. I was confused earlier thinking that Crashpad code is responsible writing the tombstones. I have the following questions.

1.If Crashpad is not enabled, then is Android responsible for capturing the renderer process crashes and writing tombstones? Is this 'fallback' currently broken and is there a bug tracking this issue?

If the Crashpad crash handler isn't enabled then the renderer process will just do the "default" thing that all android processes do and ask tombstoned/crashdumpd to generate a dump. If the crashpad handler runs for a crash in the renderer process then we just skip Android's crash handler intentionally.

As I mentioned above I am not sure if we actually *have* a way to disable crashpad any more, so it might be that we're just never getting to the point where the fallback would happen. But, it may also be broken. Even when it worked there were various weird log messages that looked like errors, though - we never got around to resolving these.

2.  Do you have guidance on where to look or what component was previously responsible for enabling tombstones for renderer processes?

Android itself does this for all processes - apps can't *enable* this because it's already enabled before their code even runs. They can *disable* it, though, by catching signals/exceptions themselves and then just not calling the default handler, which is what we do when crashpad is enabled in the renderer process.

3. How to enable Crashpad by default in our AOSP builds ? Any documentation would also help ?

You don't want to enable Crashpad; it's not going to help you unless you do a huge amount of work:

- The open source WebView code never uploads crashpad dumps anywhere as I mentioned above, so even if it's enabled it will not send you any data.

- Even if you change the code to enable uploading dumps, they will get uploaded to Google's backend which you don't have access to (and will then confuse us because they're not from our builds, so please don't)

- I don't know if there's any public/documented way to set up your own crashpad reporting backend, even if you wanted to reconfigure it to upload somewhere else, and even if you were able to set one up you would probably have to build all your own tooling for processing and understanding the dumps? I'm not really familiar with this to know the details.

So.. sorry, this seems like it might be a series of oversights on our part, but I haven't been able to test it myself. If you can share the full logs of what happens when you get a crash then that would help confirm exactly what's happening for you.

I filed https://bugs.chromium.org/p/chromium/issues/detail?id=1502654 to track checking the behavior on our end and fixing it if necessary. There are some complications around the "right" handling for renderer process crashes, though, because of onRenderProcessGone; see the bug.