Extension does not initialize after upgrade to Manifest V3... in non-US regions

[Ethan Ferrari]

On July 2 we officially upgraded from MV2 to MV3. We submitted in early-June, and after QA'ing internally with the signed extension, we published the extension update at about 11am CT on July 2. About 7pm CT, we started getting reports from our Singapore users that the application was unresponsive. Our app relies on the extension, it cannot function without it (extension is a sidecar for an internal enterprise web app). I confirmed it was published successfully and that the published ID matched the ID in our code. I also confirmed it worked correctly in the US (where we QA'd it before publishing). When I Zoomed with a support engineer in Singapore, I opened the Chrome Dev tools and attempted to call the extension manually using the `chrome.runtime.sendMessage` API. The extension was unresponsive. In an effort to debug the extension, I then opened [chrome://extensions](chrome://extensions) with the intention of inspecting the running worker code and injecting a breakpoint. However, I observed the service worker was inactive and therefore I could not inspect it. Next, I navigated to [chrome://serviceworker-internals/](chrome://serviceworker-internals/), where the service worker was not even listed. Back in chrome://extensions, I removed the extension and re-installed it, but that did not solve the issue. Ultimately, I had to roll back the extension to the previous version with MV2, and instruct all users to force update their extensions using chrome://extensions->enable "Developer mode"->Update.
I posted to https://issues.chromium.org/issues/40805401?pli=1 and sent an email to chromewebstor...@google.com requesting an extension for us on the Manifest v3 migration mandate, but was directed here.

I've been unable to find other reports of this issue. Is there anyone tracking this yet?

[Ethan Ferrari]

Apologies, bad formatting from copy/pasta in OP. Same content, less wall of text:

[woxxom]

I wonder why Chromium team allows the extensions team to push ManifestV3 despite its objectively inferior developer experience where you can't even see what's causing the registration failure of the service worker... Aren't there any adults left in the team?

[Patrick Kettner]

 Woxxom,

I understand your frustration and sympathize with the irritation,  but implying that the team is run by children is a bit much. You are always welcome to join any of the WECG calls to help steer the guidance of the platform. Your expertise and feedback would be greatly appreciated.

Regarding the actual topic of the email:

Hi Ethan!

Apologies for the slow response. Both Oliver and I have been out of the office and are catching up.

What you are seeing is consistent with a service worker registration failure. I totally agree that we need to improve the developer ergonomics dramatically for this case. In the meantime, you can see a service worker registration failure code visible on the chrome://extensions page of the extension when it is side loaded with developer mode turned on under the errors. Otherwise, try going to chrome://extension-internals on an impacted device (both when installed from the store or when side loaded). It should have some kind of state listed inside the section under your extension ID. 

Note that there could be some personal information in the extension-internals page, so if you are getting it from someone else, make sure they review it for anything sensitive before sharing.

All the best,
Patrick

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/361637e6-e80a-45af-9cdf-b7b3530ebe08n%40chromium.org.

[woxxom]

>  I understand your frustration and sympathize with the irritation,  but implying that the team is run by children is a bit much

Why are you being so literal? English is not my native language, but the expression "adult in the room" implies responsibility, not age, AFAICT. The extensions team regularly demonstrates irresponsible behavior by not fixing bugs that negatively affect developers of extensions, bugs that were reported many years ago. The bug like this one (not being able to see the cause of the registration failure) should have be a release blocker for ManifestV3 promotion.

> going to chrome://extension-internals on an impacted device (both when installed from the store or when side loaded). It should have some kind of state

This won't help much, because it won't list the actual code. I was hoping someone from your team would provide the workaround that shows the actual cause (starting chrome with --enable-logging=stderr --v=1 > log.txt 2>&1 and then looking for messages with the extension id), but I guess you didn't want to mention it, because this workaround is terrifying compared to the slick integrated debugging of a ManifestV2 background page or an event page.

>  I totally agree that we need to improve the developer ergonomics dramatically for this case.

The "positive" language was invented to let the implementer maintain the mental balance and not be overwhelmed by the incoming bug reports, but your team seems to take it too far and lets it affect the very perception the problems and consequently the prioritization.

[Cuyler Stuwe]

Co-sign that this kind of problem is severe enough and has been going on for long enough that it justifies someone having their feelings hurt a little bit.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/a2786f3f-0504-479c-986a-fe52c8970a9an%40chromium.org.

[Patrick Kettner]

I am currently at hospital and don’t have access to a version of desktop Chrome to give specific debugging steps. I appreciate you taking the time to mention more options. 

As for the nuances of English and professionalism, I am more than happy to talk outside of this thread (privately or in public), but it isn’t germane to the issue Ethan is having. 

I appreciate your passion and willingness to take the time to correct inaccuracies. I hope you voice your opinions in channels that have the best chance of causing a change you are looking for. 

[Ethan Ferrari]

Lots of unnecessary noise in this thread, let's keep it on-topic please.

Patrick, from what I understand you're suggesting I go to chrome://extension-internals on an affected device and look for some state or a code related to our extension, then share it here?

Also, any hypothesis on why it would only affect devices running in certain regions, or prefer not to speculate until seeing the output of //extension-internals?

[woxxom]

chrome://extension-internals won't help you, because it doesn't list the state when the registration of the service worker fails. To reiterate, there's no UI in Chrome to diagnose such problems in ManifestV3, hence the "noise" in this thread, sorry for hijacking it, but I don't see any way to influence the team, they've been irrationally ignoring this problem for years.

The only solution is to use the command line logging I've mentioned in my comment above: start chrome with --enable-logging=stderr --v=1 > log.txt 2>&1 and then look for messages with the extension id. Make sure you exit all chrome processes first via the browser menu -> Exit.

> any hypothesis on why it would only affect devices running in certain regions

Usually it's because of field trials that are enabled for random groups of users, sometimes geographically, so there might be a field trial that causes this problem. You can view the list in chrome://version/?show-variations-cmd

[Patrick Kettner]

Thanks for the correction, woxxom! You are right, if it is never registered it wouldn't show up there. 

Another option could be to check out chrome://histograms/#Extensions.ServiceWorkerBackground.WorkerRegistrationState on an impacted device. This will give you a histogram of the different registration states on that device. It will be various numbers, 1-25 or so, with a graph indicating the percentage of time that user is hitting any particular failure. Once you see a non 0, we can compare that to the service worker status codes inside of chromium. That should at least point out the type of failure that they are hitting. If there is any kind of pattern across your clients who are affected, we can help either with your code or with a bugfix in Chrome itself.

Appreciate you taking the time to report it!

patrick

[Ethan Ferrari]

Roger that. Our QA engineers are going to set this up tomorrow, stay tuned for results.

[Ethan Ferrari]

We have been unable to reproduce the issue now that we have a process and tools around testing it:/

Will post back here if the issue manifests again.

Thanks for everyone's help.

[Gaurang Tandon]

Hi Ethan, did your team face this issue again?

I had posted about a similar issue a month ago (link here). In short: for some users, extension service worker would become inactive after the extension update. After posting that, we published another extension update, and then we again started getting emails daily from our users having trouble due to this issue. I assume more users are affected but they didn't email us.

I look forward to help from group members or from Chrome team members regarding this issue. We are unable to push new extension updates because of it.

Thanks!

[Ryan Guilbault]

I'm also interested to hear any news/information about this since I am starting to hear reports about SWs not starting with no explanation for my clients as well. we have a chrome_debug log from a recent example that shows an Omaha request with a response indicating my extension exists and is up-to-date, but then nothing else. no context is ever created for it. other extensions do load, but much later -- so I suspect this could be a timing issue: if some sequence takes a bit too long, Chrome gives up trying to load the service worker and assumes some later action will give it a chance to be loaded. I'm debating creating a UI for my extension specifically to give the user something to attempt to interact with in the hopes it can lead to the SW getting off the ground -- it will be non-trivial work on the blind hope it could help, so not looking forward to it.

I also found this: https://issues.chromium.org/issues/336832598?pli=1 which sounds interesting. it's listed as being Citrix-specific, but it could just be that the startup of the session is similarly 'slow enough' in just the right ways as to lead to failure to launch of the SW. being otherwise in a position of complete ignorance, that is where I'm laying my bet for now.

and fwiw, I echo the sentiments about feeling powerless to debug this sort of issue and when working in enterprise environments, users are completely locked down in what they can do, not to mention they're doing their day job and not in a position to do technical debugging for us. we hear about issues hours/days after the event had occurred so the best hope we have is to collect the chrome debug logs and hope we have some breadcrumbs to look at. even just beefing up what gets put out the chrome logs would be valuable to have -- some better insights into key decision checkpoints would be very helpful, even if they're tucked into a TRACE level or whatever -- I'll opt-in when we need to. how can I submit an enhancement request for this?

[Michael Wanke]

To add to this thread and echo Ryan - we have really been struggling to understand what's going on with Chrome, Citrix, and our service worker. 

MV2 was rock solid. MV3 has been a nightmare.

I have been spending weeks with healthcare enterprises trying to understand the problem. I can watch 22 uses of Chrome in Citrix back-to-back and 2 times our service worker does not show in chrome://serviceworker-internals even though the extension shows installed (forced installed vi GPO). At times, the service worker registration eventually shows up (e.g., staring at chrome://serviceworker-internals it showed up at 3 minutes, and then another time we stopped waiting at 10 minutes).

Has anyone learned anything that helps the service worker always successful register and show in chrome://serviceworker-internals timely? 

Michael

[Jan Biniok]

> 2 times our service worker does not show in chrome://serviceworker-internals even though the extension shows installed 

FWIW: A Tampermonkey user report the same behavior at some point. The extension worked before. The only solution seems to be to disable and re-enable the extension. 

https://github.com/Tampermonkey/tampermonkey/issues/2221