Accessing Crashpad minidumps on non-rooted devices

[Brian Sheedy]

Hi all,

I'm working on getting third party Android devices set up for GPU testing, e.g. Samsung phones. This requires getting all of our testing infrastructure to work with non-rooted devices since we aren't able to root non-Google devices. One of the issues we're running into this is retrieving dumps for symbolization if something crashes during a test.

We specify where to store dumps using the `--breakpad-dump-location` flag, so you would think we could just specify a location that's readable without root permissions so we can later read/pull them. However, doing so seems to run into what I believe are SELinux permission issues, e.g. Chrome will complain that `flock` is not implemented when trying to open `settings.dat` when using the non-root directory but will work fine when using the root directory.

A few workarounds for this that we've thought of:

• Allow the storing of dumps in a different directory than `settings.dat`. This assumes that Crashpad can create minidumps fine in non-root directories and that the only actual issue is with accessing `settings.dat`. We haven't confirmed this is actually the case, but at the very least, Crashpad is capable of creating directories and `settings.dat` in the non-root directory.
• Automatically copy dumps to another location once. This has the same assumptions as workaround #1.
• Send the dump information over something like a socket. This seems iffy to me since Crashpad will be doing all of this while the process is crashing, right?

Are any of those workarounds something that would be feasible to actually add to Crashpad? If so, any pointers on where I could hack things in to see if they actually work?

Thanks,

Brian

[Joshua Peraza]

On Wed, Oct 12, 2022 at 11:05 AM 'Brian Sheedy' via Crashpad-dev <crashp...@chromium.org> wrote:

Hi all,

I'm working on getting third party Android devices set up for GPU testing, e.g. Samsung phones. This requires getting all of our testing infrastructure to work with non-rooted devices since we aren't able to root non-Google devices. One of the issues we're running into this is retrieving dumps for symbolization if something crashes during a test.

We specify where to store dumps using the `--breakpad-dump-location` flag, so you would think we could just specify a location that's readable without root permissions so we can later read/pull them. However, doing so seems to run into what I believe are SELinux permission issues, e.g. Chrome will complain that `flock` is not implemented when trying to open `settings.dat` when using the non-root directory but will work fine when using the root directory.

I think flock() is not supported by all filesystems (NFS in particular). If this was SELinux, I'd expect a message in the logcat about the call being denied. The rest of Crashpad's generic database implementation uses lock-files (files created with open + O_EXCL) to maximize compatibility with filesystems so maybe just updating the settings to also use this sort of lock-file will fix this issue. Are you otherwise able to read or write to the directory?

 

A few workarounds for this that we've thought of:

• Allow the storing of dumps in a different directory than `settings.dat`. This assumes that Crashpad can create minidumps fine in non-root directories and that the only actual issue is with accessing `settings.dat`. We haven't confirmed this is actually the case, but at the very least, Crashpad is capable of creating directories and `settings.dat` in the non-root directory.
• Automatically copy dumps to another location once. This has the same assumptions as workaround #1.
• Send the dump information over something like a socket. This seems iffy to me since Crashpad will be doing all of this while the process is crashing, right?

Are any of those workarounds something that would be feasible to actually add to Crashpad? If so, any pointers on where I could hack things in to see if they actually work?

Thanks,

Brian

--
You received this message because you are subscribed to the Google Groups "Crashpad-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to crashpad-dev...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/crashpad-dev/CAAYTr1VADY_oEnQTAqoGx_Rjro7XiNd6ZRjR%2B57FSJH9sRb%2BYg%40mail.gmail.com.

[Brian Sheedy]

> Are you otherwise able to read or write to the directory?

Crashpad can create the subdirectories (new, pending, completed, etc.) as well as the empty settings.dat, so I think yes?

> I think flock() is not supported by all filesystems (NFS in particular)

Hm, I suppose it's possible that there actually different filesystems in use, as `mount` shows a number of different filesystems in use on the device such as ext4 and sdcardfs. Can you point to an example of the lock-files usage elsewhere? I can try hacking that in.

> I'd expect a message in the logcat about the call being denied

I'm not seeing that. Here's a logcat snippet from when the settings are being initialized in case I missed something.

[Joshua Peraza]

Here's the generic database creating a lock file:

https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/client/crash_report_database_generic.cc;drc=9369044d135466db00f0abb5828f55821c0341d8;l=113

but it looks like Fuchsia is also already doing this with its settings file lock, so maybe just opt android/linux/chrome os  into fuchsia's #if branch:

https://source.chromium.org/chromium/chromium/src/+/main:third_party/crashpad/crashpad/client/settings.cc;drc=9369044d135466db00f0abb5828f55821c0341d8;l=226

[Brian Sheedy]

I hacked in a few `|| BUILDFLAG(IS_ANDROID)` in settings.h/cc and file_io_posix.cc, and it looks like we're able to successfully find and symbolize minidumps now \o/

Would it better to file a bug with this information so that someone from the Crashpad team can implement the switch? I'm not at all familiar with Crashpad, so while it seems to work in a few specific instances, I wouldn't be surprised if I ended up breaking something somewhere if I changed things myself.

[Joshua Peraza]

If you upload what you have, I'll take a look.

[Brian Sheedy]

https://chromium-review.googlesource.com/c/chromium/src/+/3952963