Chrome bug affecting ~3-5% of users of at least 4 popular extensions. Help fix it?

[Jói Sigurdsson]

Hi folks,

This is a follow-up from this older thread. After 8 weeks of myself and much of my team working to reproduce or otherwise figure out this Chrome bug, and try for workarounds in our extension to avoid it, I wrote up what we've learned so far and asked folks to help. My company is also posting two $4K bug bounties, one for a reproduction of the issue and the other for a fix.

Here is the write-up: A Chrome bug affects 3-5% of users of 5+ popular extensions. Help hunt it down!

If you can help in any way, please do. In particular I'm looking to connect with the authors of the extensions mentioned in the post (LastPass, Screencastify, Norton Password Manager, Mendeley Web Importer, CrxMouse Chrome Gestures, Save to Pocket and Mute Tab, as well as Google's own two extensions on the list) to make sure they are aware of the registered issue and can chime in there with any further details.

Cheers,

Jói

[Deco]

Interesting, based on the readup and the information provided from the timing reports, this seems like there is a race condition of some sort triggering for the setTimeout(..., 0) function which Chrome uses to cipher JS tasks which take too long to process, the bug's seemingly random nature of appearing makes this a difficult bug to test as Chrome will automatically break down tasks, itself, without user intervention. 

Chrome version 102 has support for relaxing this timing clamp, you can find it under #unthrottled-nested-timeout, in theory, if this is the culprit of what is being triggered, then relaxing the clampdown should significantly reduce the occurrences of the error reporting.

Cheers,
Decklin

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/46fe91bc-49e5-4f28-8129-15d70eb78d4dn%40chromium.org.

[PhistucK]

If I am not mistaken, since it has reportedly only started recently (a few versions back), I doubt throttling/clamping (which I believe have been there for years) are the culprit.

☆PhistucK

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CABcX4UdL0L8%2Bb%2BF0ARvnhauMTzgoAVsH9%3DT9EzOV9KzqN5KMkA%40mail.gmail.com.

[amit...@gmail.com]

Incredible writeup @Jól. I hope this Chrome bug will be prioritized and fixed!

Our users are experiencing the "extension stopped working" issue as well. We're using manifest version 2.
The browser action stops responding, no errors are seen in inspect background page while the extension is enabled but "off", and the only way we found to recover from that status is to disable and re-enable the extension (which is not something most users will do).

I starred and commented this https://bugs.chromium.org/p/chromium/issues/detail?id=1316588 

[Jói Sigurdsson]

Thanks Amit! Bringing more attention to the bug should help the Chrome team prioritize it higher.

> The browser action stops responding, no errors are seen in inspect background page while

> the extension is enabled but "off", and the only way we found to recover from that status is

> to disable and re-enable the extension (which is not something most users will do).

Sounds like exactly the same as what's happening to our users. The worst thing is that there's no way to tell them what's going on since we're not getting anything (so it just looks like they're no longer using the system)... so they simply assume it's broken and uninstall it :/ 

Here's hoping we get through this together!!

Cheers,

Jói

[Jói Sigurdsson]

Hi folks, if you know any capable Chromium freelancers who might be interested, we have now posted a job on Upwork for help tracking this issue down. Whoever is selected will be paid their hourly rate and will also remain eligible for the two $4K bounties we posted, at least until they've done $5K worth of work:

https://www.upwork.com/freelance-jobs/apply/Chromium-consultant-expert-look-elusive-timing-bug_~012592f6132a748e95/

Cheers,

Jói

[hortius tossinou9]

Thank you but I didn't understand anything!

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/6848729f-4bc6-493d-84fc-4ecd3ca932fcn%40chromium.org.

[amit...@gmail.com]

Well done Jói. 

Did anyone apply for your Upwork bounty? If not I will try to share this opportunity on some facebook groups.

Did you hear back from any of the major extensions that are affected by this issue? or from the Chromium team itself?

[Jói Sigurdsson]

Hi Amit, I have somebody working on the Upwork job I posted, but I'm up for having one or two more people working on it if they have the right skillset, so I think it'd be great if you share it.

We did not hear back from any of the popular extensions affected :/

rdevlin@ from the Chromium team has responded a few times on the issue but there's been no other communication aside from what's visible on the issue tracker :/ 

Cheers,

Jói

[Jói Sigurdsson]

The issue being discussed in this thread has achieved its 15 minutes of fame with coverage in The Register: https://www.theregister.com/2022/07/05/subtle_chromium_bug_extensions/

The reporter reached out to me yesterday wanting to send some questions but unfortunately I was out sick and didn't see their message until too late, but I think it's good coverage nonetheless and hopefully will help bring attention to the issue.

In other news we have tried again to reach out to all of the companies mentioned in the article via their support channels to see if we can get them to chime in whether this seems to be the same bug that is affecting their users. So far only one response of the "works on my computer" variety :)

Cheers,

Jói

[Uladzimir Yankovich]

As far as I understand, we're talking about a problem we discovered back in November of last year - https://bugs.chromium.org/p/chromium/issues/detail?id=1271154. When we found it, I was sure the extension platform team would mark it as a critical bug, but nothing seemed to have happened. Too bad...

[Jói Sigurdsson]

You are correct, it's either the same bug with the same underlying cause as the issue you mentioned (crbug.com/1271154), or a very similar bug for MV2 extensions, details on the MV2 bug at crbug.com/1316588

[Simeon Vincent]

Just tossing in my 2 cents, but I don't think crbug.com/1271154 and crbug.com/1316588 are the same issue. The former seems to be related to how the extension platform handles unregistering an old service and registering a new one during an extension update while the latter describes a more general failure to dispatch events to background pages. IMO both are critical issues.

IMO the slow progress on crbug.com/1316588 isn't so much an issue of prioritization as the fact that it's incredibly difficult to investigate this issue when we cannot reproduce it. To that end, I want to acknowledge and thank you, Jói, for the amount of effort you've put into trying to track this down. 

Simeon - @dotproto

Chrome Extensions DevRel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/ed5f3454-851d-47ba-bf93-9e7db28b233cn%40chromium.org.

[Jói Sigurdsson]

Thank you Simeon!

I agree it's not a given that crbug.com/1271154 and crbug.com/1316588 are related, but the reason I feel like there's still a possibility they are, is that the MV2 issue seems to only reproduce when there's a lot of the background page becoming inactive, then re-activating, which feels like it might possibly be related to the MV3 issue which as you mentioned is related to registering a new service worker. Also, in our testing and repro cases (see comments #48, 49, 50, 52), the MV3 issue would reproduce not only during an extension update but also randomly without any change incoming, just like the MV2 issue.

> IMO both are critical issues.

I agree. Any chance you might be able to convince folks to bump the priority on crbug.com/1316588 to P1? It's currently marked as a P2.

You're totally right that it's hard to fix when it's so hard to reproduce (I have been trying now for several days on a release-with-logging build of Chromium), but that shouldn't affect the priority rating, I don't think, as I believe it is affecting tons of users.

In any case, thank you Simeon, for your ongoing support of this community of extension developers. I don't know where we'd be without you!!

Cheers,

Jói

[amit...@gmail.com]

Hi @Simeon,

Thanks for helping with this. I'm joining Jói's question about potentially prioritizing the related Chrome issue to P1.

We're losing users to this bug every day, and obviously so do other extension developers which aren't necessarily aware of the reason.

If there's anything else we can do to support, let us know.

Thank you

Amit

[PhistucK]

Note that setting the priority to 1 will not magically generate an easy reproduction, this is why it cannot be 1.

☆PhistucK

[Amit Rosner]

Could be. I honestly don't know how these priorities are defined. In my opinion, difficult reproduction is a secondary factor to the number of users that suffer from unavailable service when it comes to setting priorities. Perhaps the 3,000+ currently assigned priority 1 issues are more important.

It's constantly happening on my laptop. How may I help debugging?

[PhistucK]

If you are able to build Chromium with the added-logging patch provided in the issue (or maybe Jói can provide you with an already built binary), you can run it and attach the logs to the issue once it happens.

☆PhistucK

[Jói Sigurdsson]

I do have a pre-built binary for macOS with the logging, that I can share. I am traveling today but tomorrow morning I could send a link to download the pre-built macOS files if that works for you Amit - you can respond to me privately to let me know (joi@ my company name dot com).

Jói Sigurdsson / Founder & CEO 
j...@crankwheel.com

CrankWheel 
+1 (877) 753-2945
Dalshraun 1, Hafnarfjordur 220, Iceland 
http://crankwheel.com/