Shill keeps crashing

[Sonny Sasaka]

Hi Chrome OS dev,

I have been having shill crashes for 3 times this week for 3 different devices (peppy, gandof, and caroline). Is this some known issue? If not, what's the best way to report this particular crash that I keep having?

[Mike Frysinger]

go to chrome://crashes and click "upload crashes" then see if new crashes show up in the list.  you can then look up those crashes in crash/ and see if a bug exists already, and if not, file one for the networking team to look at.

-mike

On Tue, Mar 6, 2018 at 5:52 PM, Sonny Sasaka <sonny...@chromium.org> wrote:

Hi Chrome OS dev,

I have been having shill crashes for 3 times this week for 3 different devices (peppy, gandof, and caroline). Is this some known issue? If not, what's the best way to report this particular crash that I keep having?

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Sonny Sasaka]

chrome://crashes doesn't show anything. Even if it shows something, it won't be able to upload because shill is dead and there is no network. Is there a way to upload /var/spool/crash manually? 

---
You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

--

Sonny Sasaka | Software Engineer | sonny...@google.com | +1 650-253-2040

[Kevin Cernekee]

On Tue, Mar 6, 2018 at 5:52 PM, Sonny Sasaka <sonny...@chromium.org> wrote:

I have been having shill crashes for 3 times this week for 3 different devices (peppy, gandof, and caroline). Is this some known issue? If not, what's the best way to report this particular crash that I keep having?

I would suggest attaching the logs or posting relevant snippets.

Does your crash look like https://bugs.chromium.org/p/chromium/issues/detail?id=818786 ?

2018-03-05T19:38:31.407019+00:00 INFO attestationd[1805]: Attestation service started.

2018-03-05T19:38:31.425050+00:00 ERR chapsd[1715]: TPM2Utility cannot be ready if the TPM is not owned.

2018-03-05T19:38:31.425087+00:00 WARNING chapsd[1715]: TPM initialization failed (this is expected if no TPM is available).  PKCS #11 tokens will not be available.

2018-03-05T19:38:31.434174+00:00 ERR mtpd[1793]: LIBMTP_Detect_Raw_Devices failed with 5

2018-03-05T19:38:31.435058+00:00 INFO kernel: [   15.521015] mtpd[1793]: segfault at 5bc963cabbe0 ip 00005bc963cabbe0 sp 00007ffd71d54628 error 15

2018-03-05T19:38:31.446132+00:00 INFO attestationd[1805]: Loading attestation database.

2018-03-05T19:38:31.446193+00:00 ERR attestationd[1805]: Failed to read attestation database: No such file or directory

2018-03-05T19:38:31.446208+00:00 WARNING attestationd[1805]: Creating new attestation database.

[...]

2018-03-05T19:38:33.026743+00:00 WARNING shill[2157]: [WARNING:libpolicy.cc(35)] Could not load the device policy file.

2018-03-05T19:38:33.030058+00:00 INFO shill[2157]: [INFO:chromeos_power_manager_proxy.cc(352)] OnSignalConnected interface: org.chromium.PowerManager signal: SuspendImminentsuccess: 1

2018-03-05T19:38:33.031463+00:00 INFO shill[2157]: [INFO:chromeos_power_manager_proxy.cc(352)] OnSignalConnected interface: org.chromium.PowerManager signal: SuspendDonesuccess: 1

2018-03-05T19:38:33.031767+00:00 INFO shill[2157]: [INFO:chromeos_power_manager_proxy.cc(352)] OnSignalConnected interface: org.chromium.PowerManager signal: DarkSuspendImminentsuccess: 1

2018-03-05T19:38:33.033427+00:00 ERR shill[2157]: [ERROR:crypto_des_cbc.cc(107)] Unable to load key matter from /var/lib/whitelist/owner.key

2018-03-05T19:38:33.033677+00:00 INFO shill[2157]: [INFO:key_file_store.cc(87)] Creating a new key file at /var/cache/shill/default.profile

2018-03-05T19:38:33.069861+00:00 ERR shill[2179]: *** Error in `shill': double free or corruption (!prev): 0x000058aad708de90 ***

[Sonny Sasaka]

Hi Kevin,

Yes it looks like that, but it's "memory corruption" instead of "double free or corruption". Is there any workaround at the moment?

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Kevin Cernekee]

On Tue, Mar 6, 2018 at 7:18 PM, Sonny Sasaka <sonny...@google.com> wrote:

Yes it looks like that, but it's "memory corruption" instead of "double free or corruption". Is there any workaround at the moment?

Looks like the shill crash may be a side effect of a TPM issue.  In which case the invalid access is still a bug, but not the root cause.

If the TPM error is repeatable, maybe ask Vadim or Benson to take a look at your Chromebook?

FWIW I don't see this on crash.corp.  No spike in shill crashes on recent canary builds.  But I don't know how many people are running non-test images.

[Sonny Sasaka]

Hi Kevin,

If shill keeps crashing is it possible to still upload the crash reports when network doesn't work? 

--
--
Chromium OS Developers mailing list: chromiu...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Mike Frysinger]

On Tue, Mar 6, 2018 at 8:42 PM, 'Sonny Sasaka' via Chromium OS dev <chromiu...@chromium.org> wrote:

On Tue, Mar 6, 2018 at 7:39 PM, Kevin Cernekee <cern...@chromium.org> wrote:

On Tue, Mar 6, 2018 at 7:18 PM, Sonny Sasaka <sonny...@google.com> wrote:

Yes it looks like that, but it's "memory corruption" instead of "double free or corruption". Is there any workaround at the moment?

Looks like the shill crash may be a side effect of a TPM issue.  In which case the invalid access is still a bug, but not the root cause.

If the TPM error is repeatable, maybe ask Vadim or Benson to take a look at your Chromebook?

FWIW I don't see this on crash.corp.  No spike in shill crashes on recent canary builds.  But I don't know how many people are running non-test images.

If shill keeps crashing is it possible to still upload the crash reports when network doesn't work? 

the crashes are all queued locally until the network comes back up.  so reboot the system and when the network comes up, trigger a crash upload.

if you can't get the network up, then there's no way to access the crashes directly currently.  you could try connecting a USB ethernet dongle.

-mike

[Kevin Cernekee]

On Tue, Mar 6, 2018 at 11:15 PM, Mike Frysinger <vap...@chromium.org> wrote:

FWIW I don't see this on crash.corp.  No spike in shill crashes on recent canary builds.  But I don't know how many people are running non-test images.

If shill keeps crashing is it possible to still upload the crash reports when network doesn't work? 

the crashes are all queued locally until the network comes back up.  so reboot the system and when the network comes up, trigger a crash upload.

if you can't get the network up, then there's no way to access the crashes directly currently.  you could try connecting a USB ethernet dongle.

We can check out the crashes in person... I think Sonny was speculating as to why this error didn't appear on crash.corp.

My original assumption was that it only happened after logon, but looking at the logs again, it seems to happen immediately after boot.  So the system probably wouldn't have a chance to upload the reports.  We should definitely try to make shill more resilient to this failure if possible.

[Mike Frysinger]

based on your "non-test image" comment, i'm assuming Sonny is running an official image and not in dev mode in which case, it's not possible to extract/view crash reports directly.  putting the device into dev mode would wipe the reports.

if he's already in dev mode, then yeah, extracting the crashes onto a USB stick would be trivial.

if it was a locally built developer image, then the crash reports wouldn't be uploaded in the first place, just queued locally and left there forever.

-mike

[Drew Wilson]

Probably a coincidence, but we have reports of devices with TPM problems leading to stateful partition wipe happening upon upgrade to M63: https://bugs.chromium.org/p/chromium/issues/detail?id=810716.

Not sure if there's something more widespread happening with TPMs.

-atw

--

[Sonny Sasaka]

I was using a test image so I can easily access /var/spool/crash. But reboot or power cycle won't bring the network back. I already had this 3 times and I had to reflash the device and setup all my environment for testing all over again. It will be good to have a workaround so I don't have to drain productivity by reflashing the device every time I encounter this. So I have the files in /var/spool/crash. How can I interpret this into something meaningful?

---
You received this message because you are subscribed to the Google Groups "Chromium OS dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-os-dev+unsubscribe@chromium.org.

[Mike Frysinger]

a test image won't upload crashes (by design) even if you had network access

is it an image you built locally or fetched from the server ?  if the former, even if you uploaded the crash reports, you wouldn't get symbols because the server wouldn't know any of the modules.

this page has some various tips you can try to symbolize the crashes locally:

  http://dev.chromium.org/chromium-os/packages/crash-reporting

-mike

--

Sonny Sasaka | Software Engineer | sonnysasaka@google.com | +1 650-253-2040

[Stéphane Marchesin]

+djmk@ since he was bisecting shill for this problem (some VMs lose network because of that).

[Sonny Sasaka]

It's image from server. Thanks for the doc! Will take a look at that. Sadly I didn't find this doc when Googling "chrome os debug symbols".

Sonny Sasaka | Software Engineer | sonny...@google.com | +1 650-253-2040

[Sonny Sasaka]

I tried following the doc and it mentions http://chromeos-images.corp.google.com/ to get debug symbols. However that page shows Error 502 (ÜberProxy something). Is this a known issue? Or maybe I need more permission?

[Kirtika Ruchandani]

@Sonny:

1. for debugging a crash report, does this help? https://www.chromium.org/chromium-os/packages/crash-reporting/debugging-a-minidump

2. for downloading images, you want a link that starts with .. https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/

I usually build up the URL by starting with a known bookmark and replacing the builder/release version: 

e.g. https://pantheon.corp.google.com/storage/browser/chromeos-image-archive/meowth-release/R67-10461.0.0

3. You mentioned you are seeing this on a caroline and gandof. Can you provide exact details of device, OS version and repro steps? I tried reproing on ToT on my cyan and meowth (the original bug reporter saw it on a coral variant),

but couldnt reproduce the crash. 

Do you have a last known good version?

Sonny Sasaka | Software Engineer | sonnysasaka@google.com | +1 650-253-2040

[Sonny Sasaka]

Hi Kirtika,

Thanks for the information. I am able to get the debug symbols from the link you provided!

By the way, Kevin (cerneeke@) came by and was able to help me bring the network up so I could try reverting all the changes I made. Turns out my modified libchrome & libbrillo wasn't binary-compatible with the other binaries including shill. So, mysteries solved. Thank you for all the help!

[Mike Frysinger]

heh, chromeos-images.corp.google.com is an old system we turned down years ago.  i'm not seeing that listed anywhere in the public docs, so i'm guessing you're looking at an internal one.  can you link to it so i can update it ?

in the meantime, Kirtika listed the current alternatives (grabbing the files from the chromeos-image-archive GS bucket).

-mike

Sonny Sasaka | Software Engineer | sonnysasaka@google.com | +1 650-253-2040

[Sonny Sasaka]

I got the link from the doc you gave me:

http://dev.chromium.org/chromium-os/packages/crash-reporting

Then it links to http://dev.chromium.org/chromium-os/packages/crash-reporting/debugging-a-minidump, then to https://sites.google.com/a/google.com/mkrebs/references/chrome-os#TOC-Setup-files-for-debugger, then to https://sites.google.com/a/google.com/chromeos/resources/engineering/crash-reporting-faq#TOC-Where-can-I-find-the-symbols-for-a-particular-Chrome-OS-version-

Sonny Sasaka | Software Engineer | sonny...@google.com | +1 650-253-2040

[Mike Frysinger]

i've updated the chromeos pages, but don't have control over the mkrebs pages

-mike

Sonny Sasaka | Software Engineer | sonnysasaka@google.com | +1 650-253-2040

[Sonny Sasaka]

Thanks for fixing the doc!

Sonny Sasaka | Software Engineer | sonny...@google.com | +1 650-253-2040