Extension does not initialize after upgrade to Manifest V3... in non-US regions

[Ethan Ferrari]

On July 2 we officially upgraded from MV2 to MV3. We submitted in early-June, and after QA'ing internally with the signed extension, we published the extension update at about 11am CT on July 2. About 7pm CT, we started getting reports from our Singapore users that the application was unresponsive. Our app relies on the extension, it cannot function without it (extension is a sidecar for an internal enterprise web app). I confirmed it was published successfully and that the published ID matched the ID in our code. I also confirmed it worked correctly in the US (where we QA'd it before publishing). When I Zoomed with a support engineer in Singapore, I opened the Chrome Dev tools and attempted to call the extension manually using the `chrome.runtime.sendMessage` API. The extension was unresponsive. In an effort to debug the extension, I then opened [chrome://extensions](chrome://extensions) with the intention of inspecting the running worker code and injecting a breakpoint. However, I observed the service worker was inactive and therefore I could not inspect it. Next, I navigated to [chrome://serviceworker-internals/](chrome://serviceworker-internals/), where the service worker was not even listed. Back in chrome://extensions, I removed the extension and re-installed it, but that did not solve the issue. Ultimately, I had to roll back the extension to the previous version with MV2, and instruct all users to force update their extensions using chrome://extensions->enable "Developer mode"->Update.
I posted to https://issues.chromium.org/issues/40805401?pli=1 and sent an email to chromewebstor...@google.com requesting an extension for us on the Manifest v3 migration mandate, but was directed here.

I've been unable to find other reports of this issue. Is there anyone tracking this yet?

[Ethan Ferrari]

Apologies, bad formatting from copy/pasta in OP. Same content, less wall of text:

[woxxom]

I wonder why Chromium team allows the extensions team to push ManifestV3 despite its objectively inferior developer experience where you can't even see what's causing the registration failure of the service worker... Aren't there any adults left in the team?

[Patrick Kettner]

 Woxxom,

I understand your frustration and sympathize with the irritation,  but implying that the team is run by children is a bit much. You are always welcome to join any of the WECG calls to help steer the guidance of the platform. Your expertise and feedback would be greatly appreciated.

Regarding the actual topic of the email:

Hi Ethan!

Apologies for the slow response. Both Oliver and I have been out of the office and are catching up.

What you are seeing is consistent with a service worker registration failure. I totally agree that we need to improve the developer ergonomics dramatically for this case. In the meantime, you can see a service worker registration failure code visible on the chrome://extensions page of the extension when it is side loaded with developer mode turned on under the errors. Otherwise, try going to chrome://extension-internals on an impacted device (both when installed from the store or when side loaded). It should have some kind of state listed inside the section under your extension ID. 

Note that there could be some personal information in the extension-internals page, so if you are getting it from someone else, make sure they review it for anything sensitive before sharing.

All the best,
Patrick

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/361637e6-e80a-45af-9cdf-b7b3530ebe08n%40chromium.org.

[woxxom]

>  I understand your frustration and sympathize with the irritation,  but implying that the team is run by children is a bit much

Why are you being so literal? English is not my native language, but the expression "adult in the room" implies responsibility, not age, AFAICT. The extensions team regularly demonstrates irresponsible behavior by not fixing bugs that negatively affect developers of extensions, bugs that were reported many years ago. The bug like this one (not being able to see the cause of the registration failure) should have be a release blocker for ManifestV3 promotion.

> going to chrome://extension-internals on an impacted device (both when installed from the store or when side loaded). It should have some kind of state

This won't help much, because it won't list the actual code. I was hoping someone from your team would provide the workaround that shows the actual cause (starting chrome with --enable-logging=stderr --v=1 > log.txt 2>&1 and then looking for messages with the extension id), but I guess you didn't want to mention it, because this workaround is terrifying compared to the slick integrated debugging of a ManifestV2 background page or an event page.

>  I totally agree that we need to improve the developer ergonomics dramatically for this case.

The "positive" language was invented to let the implementer maintain the mental balance and not be overwhelmed by the incoming bug reports, but your team seems to take it too far and lets it affect the very perception the problems and consequently the prioritization.

[Cuyler Stuwe]

Co-sign that this kind of problem is severe enough and has been going on for long enough that it justifies someone having their feelings hurt a little bit.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/a2786f3f-0504-479c-986a-fe52c8970a9an%40chromium.org.

[Patrick Kettner]

I am currently at hospital and don’t have access to a version of desktop Chrome to give specific debugging steps. I appreciate you taking the time to mention more options. 

As for the nuances of English and professionalism, I am more than happy to talk outside of this thread (privately or in public), but it isn’t germane to the issue Ethan is having. 

I appreciate your passion and willingness to take the time to correct inaccuracies. I hope you voice your opinions in channels that have the best chance of causing a change you are looking for. 

[Ethan Ferrari]

Lots of unnecessary noise in this thread, let's keep it on-topic please.

Patrick, from what I understand you're suggesting I go to chrome://extension-internals on an affected device and look for some state or a code related to our extension, then share it here?

Also, any hypothesis on why it would only affect devices running in certain regions, or prefer not to speculate until seeing the output of //extension-internals?

[woxxom]

chrome://extension-internals won't help you, because it doesn't list the state when the registration of the service worker fails. To reiterate, there's no UI in Chrome to diagnose such problems in ManifestV3, hence the "noise" in this thread, sorry for hijacking it, but I don't see any way to influence the team, they've been irrationally ignoring this problem for years.

The only solution is to use the command line logging I've mentioned in my comment above: start chrome with --enable-logging=stderr --v=1 > log.txt 2>&1 and then look for messages with the extension id. Make sure you exit all chrome processes first via the browser menu -> Exit.

> any hypothesis on why it would only affect devices running in certain regions

Usually it's because of field trials that are enabled for random groups of users, sometimes geographically, so there might be a field trial that causes this problem. You can view the list in chrome://version/?show-variations-cmd

[Patrick Kettner]

Thanks for the correction, woxxom! You are right, if it is never registered it wouldn't show up there. 

Another option could be to check out chrome://histograms/#Extensions.ServiceWorkerBackground.WorkerRegistrationState on an impacted device. This will give you a histogram of the different registration states on that device. It will be various numbers, 1-25 or so, with a graph indicating the percentage of time that user is hitting any particular failure. Once you see a non 0, we can compare that to the service worker status codes inside of chromium. That should at least point out the type of failure that they are hitting. If there is any kind of pattern across your clients who are affected, we can help either with your code or with a bugfix in Chrome itself.

Appreciate you taking the time to report it!

patrick

[Ethan Ferrari]

Roger that. Our QA engineers are going to set this up tomorrow, stay tuned for results.