Chrome 76 + 78 - Extension Corrupted Randomly

[Austin Wang]

Hi there, we have an extension with about 28k installed users. On Friday/Today we've received a spike in reports of our extension getting disabled by Chrome and being marked as corrupted.

- Users have the option to repair, which seems to work, but then it gets corrupted again shortly after (like within an hour)

- We have received reports from more than 10 people today, where usually we get < 1 issue like this per month

- We have not uploaded or published any extension updates in the last 7 days

- Doesn't seem to be affected by any specific extension action, it just breaks randomly it seems as far as we can tell so far.

- We have an employee who is affected and it happens to her every 15 - 20 minutes after repair. If diagnostic is needed we can run any command or whatever.

- We tried installing the canary version of Chrome on her computer to solve this issue and it didn't solve it

Specific versions we've observed this on so far

76.3809.87

78.03874.3

 

[Austin Wang]

I also did see there were 1-2 other corruption related issues from a few weeks back but those were supposed to have been resolved already where as this seems very new.

[Eric Lawrence]

The fix for biggest known break I've seen (https://crbug.com/977805) landed in the following versions:

• 76.0.3809.74
• 77.0.3852.0




Is your extension this one? https://chrome.google.com/webstore/detail/groove-for-gmail/fdkgjhmihbfpjgodmbmmmgmgnmemokda?hl=en-US

If you run Chrome with

    chrome --enable-logging --v=1

It will generate a chrome_debug.log in your profile directory. At the point the extension is disabled it will note the failure, look for chrome_content_verifier_delegate in the log.

[Eric Lawrence]

Are you seeing this only on one OS, or across both Mac and Windows?

Taking a look at your extension package, one unusual thing I see is that you've got two images: "images/ .png" and "images/ @2x.png", where the filename contains a space character. While I can't see why this would necessarily pose a problem, I don't see these resources getting directly used within the extension; is their presence intentional?

[Vijay Pemmaraju]

Hi Eric,

One of our internal users was affected by this, so I had her run chrome with logging, and I found these two lines in her chrome_debug.log:

[4558:35587:0807/142558.607005:VERBOSE1:content_verify_job.cc(274)] job failed for fdkgjhmihbfpjgodmbmmmgmgnmemokda css/fonts/fontawesome-webfont.ttf reason:3

[4558:775:0807/142558.607150:VERBOSE1:content_verifier.cc(520)] VerifyFailed fdkgjhmihbfpjgodmbmmmgmgnmemokda reason:3

These seem to be the culprit as immediately after, there's several instances of this:

[11392:775:0807/142558.759320:VERBOSE1:script_context.cc(126)] Destroyed context for extension

extension id: fdkgjhmihbfpjgodmbmmmgmgnmemokda

effective extension id: fdkgjhmihbfpjgodmbmmmgmgnmemokda

Let me know if that's helpful.

- Vijay

On Monday, August 5, 2019 at 1:42:39 PM UTC-7, Austin Wang wrote:

[Eric Lawrence]

This is indeed helpful. "Reason: 3" is defined as:

  // Some of the content read did not match the expected hash.
    HASH_MISMATCH

So this suggests that the resource was indeed found in the package (correct URL mapping) but when the content was read from the disk, it wasn't as expected.

• Can you confirm the exact Chrome version number for this repro case?
• Can you verify that the fontawesome-webfont.ttf file on her disk is byte-for-byte identical with the expected file? 

[Vijay Pemmaraju]

• The chrome version was 76.3809.87.
• I ran a cmp against the ttf file on her machine and the one we bundle into the extension and they're exactly the same byte-for-byte, as far as I can tell.
  

[Vijay Pemmaraju]

Hi Eric,

We pushed an update on Friday removing that file, but it appears that the corruption is still happening on the latest version. We haven't gathered logs yet for the newer build, but I'm guessing that the hash verification is now failing for some other file. Is there anything we can do to diagnose this issue other than waiting the corruption to happen and collecting the debug.log file? Is there any way to run the content verification on demand?

- Vijay

[PhistucK]

Is this happening to everyone, or just to some users?

For the latter, perhaps some security software is modifying the file.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/c0e71496-322e-488d-a162-7f0643292af0%40chromium.org.

[Eric Lawrence]

Alas, I don't know any way to "force" verification for all resources. 

If your extension corruption is happening for web accessible resources, then you could build a simple HTML page that pulls in every resource from the web_accessible_resources section and that would cause an immediate failure if any of the hashes had changed. 

Having said that, in most of the places we've checked, developers are saying that the files on disk and in the original package are byte-for-byte identical, which would point to a code bug in the verifier logic rather than external modification.

[PhistucK]

Can security software modify the file only by interception rather than on disk?

I mean, when applications read a file, the security software intervenes and returns their version of the bytes of the file. Is this even possible?

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/b2417896-ac0d-4457-bbd3-599593653fc2%40chromium.org.

[Vijay Pemmaraju]

We've heard back from a few customers who are experiencing the issue, and they said they have no antivirus/security software installed.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.

[Austin Wang]

The antivirus theory also doesn't match with the fact that this issue has a very distinct start time, which is (August 3rd - 5th) where as antivirus would be a persistent problem. We are getting 15+ reports of this per day where before we got 0.

[PhistucK]

Well, anti-virus definitions can be/are being updated daily (unless you disable this feature), so the theory could still apply... This is probably not the case, but, just noting.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/b2417896-ac0d-4457-bbd3-599593653fc2%40chromium.org.

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/9ddafe13-e786-4e75-86b8-59dc57b72a72%40chromium.org.

[Gildas]

Some using SingleFile on Ubuntu reported me exactly the same issue. I also see in my dashboard an unusual peak of uninstalls.

[CRIMX]

I also own a 60k extension Saladict that also suffers from the random corruption issue on Chrome >= 76. More and more users report issue everyday. Mostly on Mac and Linux. I had a user send me the log and the installed extension files.  The "corrupted" file according to the log is identical with the one I built. Pretty sure it's Chrome's bug.

[CRIMX]

FYI, the corrupted file is a css file loaded from a injected iframe. Something like

<iframe
  srcdoc="<!DOCTYPE html><html><head><link type="text/css" rel="stylesheet" href="chrome-extension://[extension-id]/panel.css" /></head></html>"
>
</iframe>

This works well before Chrome 76.

[CRIMX]

:(

[Austin Wang]

@Eric Lawrence - Can we get some more attention on this issue? We've collected a dozen log files and are now moving files from extension to CDN in attempt to reduce files that could theoretically corrupt. 

Now that we have more developers reporting the same issue - it seems more and more likely that this is a chrome bug. What do we need to do in order to get more attention on this and move forward toward diagnosis / fix?

On Friday, August 16, 2019 at 12:30:48 AM UTC-7, CRIMX wrote:

:(

[Markus Hartung]

I would suggest you comment on https://bugs.chromium.org/p/chromium/issues/detail?id=977805 and try to get it reopened or file a new bug.

[Simeon Vincent]

I think filing a new bug is the best route as it may not be the same underlying issue. When doing so, please include a link to your extension's listing in the web store and attach the CRX. You may also want to link to https://crbug.com/977805 in your report. 

Simeon - @dotproto

Extensions Developer Advocate

[PhistucK]

Simeon, it would be great if you talked to the team about this class of issues and have them prioritize them above any other work.

It is not the first time that an extension (as well as the user or system or other applications) does nothing wrong (I am not sure how the extension could do anything wrong) and the verifier declares it as corrupted.

This verifier must be bug-free (to a reasonable extent), otherwise it is like getting a non-pirated product only to find that it is unplayable because the DRM method used by it made it unreadable to the player.

I feel like the verifier is bug-full. One prominent bug that comes to mind was the any-file-that-weighs-a-multiple-of-4096-is-corrupted. This kind of bugs is disrespectful to the users and to the extension authors.

An extension system is useless if it cannot run extensions.

☆PhistucK

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/db9fce9d-7f44-4499-9cec-c930829a61b9%40chromium.org.

[Austin Wang]

Completely agree. 

I've filed a new bug as suggested.

https://bugs.chromium.org/p/chromium/issues/detail?id=995436

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.

[Simeon Vincent]

Prioritization isn't exactly my responsibility, but I can certainly make noise. That said, I've got an engineer sitting to my left looking at the issue now :)

I posted a couple requests for information in bug report (995436). I also just filed 995554 to request that Chrome expose a content verification tool to assist in cases such as this.

Simeon - @dotproto

Extensions Developer Advocate

[Eric Lawrence]

The primary extension developer has found a new cause for this problem: https://bugs.chromium.org/p/chromium/issues/detail?id=995436#c21

ContentVerifyJob::OnHashesReady -> FinishBlock doesn't take has_ignorable_read_error_ into conderation. So, if hashes become ready after some benign read errors happen, we hit this bug.

This bug is now marked "Started" although I don't yet see a CL to fix it.

To simplify triggering this bug and bugs like it, I built out a simple page: https://webdbg.com/test/window/. If you visit this page and disable your popup blocker, it will open (and close) a flood of tabs and you can wait to hit a failure.

[Eric Lawrence]

A fix for `Chromium browser extensions incorrectly marked as "Corrupt"` landed tonight. Likely to appear in 78.0.3894 tomorrow: https://chromium.googlesource.com/chromium/src.git/+/05f3ae039c2f3c739b1c09219c08028b4a5661ff

Most likely, after this fix has been verified in Canary, it'll be backported to 76 and 77.

[Eric Lawrence]

The fix appears to have missed the cut for 78.0.3894, so it'll probably appear in 78.0.3895. If you'd like to track the fix for Canary (and any branches to which it's merged), you can look here: https://storage.googleapis.com/chromium-find-releases-static/05f.html#05f3ae039c2f3c739b1c09219c08028b4a5661ff

[PhistucK]

A nicer looking front end for this can be found here -

https://chromiumdash.appspot.com 

☆PhistucK

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/8a5851d0-8b97-46cb-81b7-81abc32cedb2%40chromium.org.