declarativeNetRequest extension - error on update

[Browser Extenstion]

I have an ad blocker using declarativeNetRequest.

const defaultRulesSet = ['ruleset_1', 'ruleset_2', 'ruleset_3', 'ruleset_4', 'ruleset_5', 'ruleset_6', 'ruleset_7', 'ruleset_8', 'ruleset_9', 'ruleset_10'];
chrome.declarativeNetRequest.updateEnabledRulesets({
disableRulesetIds : [defaultRulesSet.pop()]
})
chrome.declarativeNetRequest.updateEnabledRulesets({
enableRulesetIds : [defaultRulesSet.pop()]
})

Due to it being an ad blocker, the list of rules for blocking is indeed extensive.

If I'm installing the extension for the first time, there are no issues.
If I'm updating (reload button) the local version during development, there are no issues.

The problem occurs when the extension is updated from the store: an extension error appears in the top right corner of the browser.
If I check the current sets after update (with the error)
chrome.declarativeNetRequest.getEnabledRulesets(rules => console.log(rules)) // result: only 2-3 random rulesets are enabled

If I'm trying to enable any other ruleset I get the Unchecked runtime.lastError: Internal error.
I want to note right away that it's not a rate limit error.

From my observations, the error only occurs if the extension is updated after the browser is launched.
In other words, if the browser is already running, and the extension updates, there are no problems.
However, if the browser was closed and the extension updates upon reopening, the error occurs.

I can't understand why even in manual mode, I'm getting an error (like in the screenshot):

[Browser Extenstion]

In the previous message I wrote the not corret code
chrome.declarativeNetRequest.updateEnabledRulesets({
disableRulesetIds : [defaultRulesSet.pop()]
})

The correct code like it is in the extension:

let rules = [...defaultRulesSet];
while (rules.length > 0) {
chrome.declarativeNetRequest.updateEnabledRulesets({
enableRulesetIds : [rules.pop()]

[Browser Extenstion]

Some additional information from chrome://serviceworker-internals/?devtools

Console: {"lineNumber":6,"message":"Uncaught (in promise) Error: Internal error.","message_level":3,"sourceIdentifier":1,"sourceURL":"chrome-extension://_ext_id_/sw.js"}

First, I don't have line 6 in the service worker. So I don't really know what the error is talking about.

Anyone from Google?

[Browser Extenstion]

Another addition:

If I download the CRX file from the Chrome Web Store and drag it onto the chrome://extensions page (regardless of whether an old version or the current one from the store is installed), everything starts working again without errors.

So, errors occur specifically when updating from the store.

Any help would be appreciated.

[Browser Extenstion]

And one more note:

If I click update button on the chrome://extensions page  - the extension updated without errors.

[wOxxOm]

It looks like a bug in the browser - it ignores randomly subsequent updateEnabledRulesets while it still processes the previous one. Try using the API the way it was intended to be used, i.e. just a single call:

chrome.declarativeNetRequest.updateEnabledRulesets({ enableRulesetIds : rules })

[Browser Extenstion]

I tried the option you suggested. In this case, I still get the same error.

Using my method, there is some chance that part of the lists will be loaded. 

When passing the entire list at once, I get an error, and consequently, none of the rulesets are updated.

>>> It looks like a bug in the browser

The problem occurs on different browsers and platforms (Mac, Windows).

At the moment, I don't quite understand how to debug this issue either.
The error I'm receiving is not informative at all. 

The only idea that has come to my mind so far is to create a private extension, upload it to Chrome, and attempt various update options until a solution is found.

[wOxxOm]

A bug in the browser usually means it's a bug in the source code of Chromium, which is used in all Chromium-based browsers. I vaguely remember someone reported a similar problem were  declarativeNetRequest couldn't properly index the rules after an update. A possible workaround would be to call updateEnabledRulesets several times with an increasing delay, until all rulesets are updated. Note though you must wait for the previous call to complete first:

chrome.runtime.onInstalled.addListener(async info => {

  for (const id of defaultRulesSet) {

    let delay = 50;

    while (true) {

      try {

        await chrome.declarativeNetRequest.updateEnabledRulesets({ enableRulesetIds: [id] });

        break;

      } catch (e) {

        await new Promise(r => setTimeout(r, delay));

        delay *= 1.25;

      }

    }

  }

});

[Browser Extenstion]

Thanks, I'll try.

The problem at the moment is that after the error, I can't even update a single ruleset separately:
chrome.declarativeNetRequest.updateEnabledRulesets({
    enableRulesetIds: ['ruleset_2']
})

The result is Internal error in promise.

But I will create a separate extension to test this code during updates.

[Oliver Dunk]

Hi,

Thanks so much for sharing this (and thanks wOxxOm for the help up to this point). I would never expect to see an internal error like that and so it would definitely be worth opening a bug at https://crbug.com/ if you haven't already.

Creating a simple reproduction extension would definitely be helpful. You may also find the update testing tool to be useful which allows you to test the update flow without submitting to the web store: https://github.com/GoogleChromeLabs/extension-update-testing-tool

To confirm, do you only see this when enabling the rulesets individually? Could you make a single call with an array containing all of the rulesets you want to enable instead?

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/4f3b8b81-0267-4c86-a64b-5f24a541c265n%40chromium.org.

[Browser Extenstion]

>>> Thanks so much for sharing this (and thanks wOxxOm for the help up to this point). I would never expect to see an internal error like that and so it would definitely be worth opening a bug at https://crbug.com/ if you haven't already.

I will try to create a bug report, but it will be very difficult to show the error to the Google team. 

For this, I would need them to install the version from the store and then wait for an update to that version. 

Unfortunately, I don't think anyone will go through such maneuvers.

>>> Creating a simple reproduction extension would definitely be helpful. You may also find the update testing tool to be useful which allows you to test the update flow without submitting to the web store: https://github.com/GoogleChromeLabs/extension-update-testing-tool

Unfortunately, this solution doesn't work in my case.
As I described earlier, when I press the "update" button, the extension updates without errors. 

The error occurs when the browser automatically updates the extension.

>>> To confirm, do you only see this when enabling the rulesets individually? Could you make a single call with an array containing all of the rulesets you want to enable instead?

It doesn't matter how you update the ruleset. 

If you try to update the entire array, in that case, all rulesets will be inactive. 

If you update them one by one (extracting them from the array), there is a chance that some rulesets will be updated correctly.

And the biggest problem is that if you open the service worker for the extension where the error occurred and try to update the rules or all rules at once, there will always be an Internal error.

Browser restart or chrome.runtime.reload() does not help in this case.

[Oliver Dunk]

Thanks for the reply.

Even if you don't have a reproduction, opening a bug and sharing the general issue and that you are seeing an internal error would be great.

Given your description, I'm surprised we haven't seen this issue reported in the past. That makes me wonder if something about your setup or the specific rules you are trying to use is causing this.

Unfortunately, this solution doesn't work in my case.
As I described earlier, when I press the "update" button, the extension updates without errors. 
The error occurs when the browser automatically updates the extension.

Chrome sees extensions using this tool as very similar to extensions from the web store. You should be able to test the automatic update process using it, albeit having to wait for that to happen. Calling chrome.runtime.requestUpdateCheck() may help unless this affects how the bug behaves.

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/ea9475db-56ff-4171-9c61-4e9ba4201163n%40chromium.org.

[Browser Extenstion]

First of all, Oliver, a big thank you for the tool.
I don't know how much time I would have needed to test the extension without it.

Now, onto the actual question. I'll try to describe the issue as structured as possible.

I created three options for testing.
For each of the options, we have an initial version 1 and an updated version 2.
We will be testing using the tool that Oliver suggested.
All extensions have a structure of two rulesets (but different initial options):
"declarative_net_request": {
"rule_resources": [
{
"id": "ruleset_1",
......
},
{
"id": "ruleset_2",
......
}
]
}

1. The first option - all rulesets are enabled in both versions
"declarative_net_request": {
"rule_resources": [
{
"id": "ruleset_1",
"enabled": true,
"path": "rules/ruleset_1.json"
},
{
"id": "ruleset_2",
"enabled": true,
"path": "rules/ruleset_2.json"
}
]
}

AB_v_1 - each ruleset has two rules
AB_v_2 - we remove one rule from ruleset_1

After the update, all rulesets are enabled (as they should be), but we have an error as shown below:

2. The second option - one ruleset are enabled in both versions
"declarative_net_request": {
"rule_resources": [
{
"id": "ruleset_1",
"enabled": false,
"path": "rules/ruleset_1.json"
},
{
"id": "ruleset_2",
"enabled": true,
"path": "rules/ruleset_2.json"
}
]
}

AB_v_1 - each ruleset has two rules
AB_v_2 - we remove one rule from ruleset_1

After the update: the same behavior as in the first option, with the same error.

3. The third option - all rulesets are disabled in both versions
"declarative_net_request": {
"rule_resources": [
{
"id": "ruleset_1",
"enabled": false,
"path": "rules/ruleset_1.json"
},
{
"id": "ruleset_2",
"enabled": false,
"path": "rules/ruleset_2.json"
}
]
}

AB_v_1 - each ruleset has two rules
AB_v_2 - we remove one rule from ruleset_1

After the update, we get an Internal Error, and only ruleset_2 (the one that didn't undergo changes in the updated version) will be active. The ruleset_1 that was modified won't load.




I have created simple extensions for each of the scenarios to demonstrate how it looks. Since the first and second options behave the same way, I will only upload one of the variations. And a separate variation for the third option.
Steps to reproduce:
1. Open the tool
2. Upload V_1 extension
3. Download the crx and drag and drop it to the chrome://extensions page
4. Turn on "Collect errors" button
5. Upload V_2 extension to the tool
6. Open service worker and run chrome.runtime.requestUpdateCheck()


------------------------------------------------------------------------------------------------------------------
Now I have two questions:
1. I couldn't find any specific information in the declarativeNetRequest documentation indicating that at least one ruleset must always be enabled in the manifest.
Is there such a limitation?

2. Cannot load extension with file or directory name _metadata. Filenames starting with "_" are reserved for use by the system.
Is this error related to the fact that we are using the tool during the update, or will it also occur when updating from the Chrome Web Store?
I see the bug https://crbug.com/1061292 from March 2020.
But according to the https://groups.google.com/a/chromium.org/g/chromium-extensions/c/PP4RI98I_Vo/m/FN-vfKEmDwAJ thread the problem still exists.

[Browser Extenstion]

Sample extensions for the first scenario - all rulesets are enabled in initial and updated versions:

Change file name extension from crx to zip
Follow the steps:

[Browser Extenstion]

Sample extensions for the third scenario - all rulesets are disabled in initial and updated versions:

Change file name extension from crx to zip
Follow the steps:

1. Open the tool
2. Upload V_1 extension
3. Download the crx and drag and drop it to the chrome://extensions page
4. Turn on "Collect errors" button
5. Upload V_2 extension to the tool
6. Open service worker and run chrome.runtime.requestUpdateCheck()

[Oliver Dunk]

Hi,

Thanks so much for all of the time investigating here - and sorry for the slow reply, a number of us have been travelling and so I'm slightly behind on a few threads like this one.

My understanding based on what you said is that loading and install the v1 of the third scenario with the tool, and then updating to v2 should produce the internal error - is that right? I gave that a try but wasn't able to.

If that is the case, please do file a bug at https://crbug.com/ with the extensions and steps to reproduce. That way the engineering team can take a look and they may have more of an idea as to what could be happening.

On your questions:

I couldn't find any specific information in the declarativeNetRequest documentation indicating that at least one ruleset must always be enabled in the manifest.

I'm not aware of anything like this. Were you having problems?

Cannot load extension with file or directory name _metadata. Filenames starting with "_" are reserved for use by the system.

Is this error related to the fact that we are using the tool during the update, or will it also occur when updating from the Chrome Web Store?

Internally, Chrome creates an _metadata folder when you load a folder as an unpacked extension. This stores some things like optimized DNR rules. If you accidentally include this folder in an upload to the tool, it can cause that warning. I think it would also affect the web store too. Making sure it doesn't exist in whatever zip you upload should prevent that.

Hope this helps!

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Browser Extenstion]

>>> My understanding based on what you said is that loading and install the v1 of the third scenario with the tool, and then updating to v2 should produce the internal error - is that right? I gave that a try but wasn't able to.

Follow the steps:

1. Open the tool
2. Upload V_1 extension
3. Download the crx and drag and drop it to the chrome://extensions page
4. Turn on "Collect errors" button
5. Upload V_2 extension to the tool

6. Open service worker and run chrome.runtime.requestUpdateCheck() <-- That's an essential condition. You do NOT need click the update button at any point.

>>> I'm not aware of anything like this. Were you having problems?

This is exactly what I'm referring to in my previous messages.

The internal error occurs only when all of the rulesets are in the "enabled":  false state.

>>> Internally, Chrome creates an _metadata folder when you load a folder as an unpacked extension. This stores some things like optimized DNR rules. If you accidentally include this folder in an upload to the tool, it can cause that warning. I think it would also affect the web store too. Making sure it doesn't exist in whatever zip you upload should prevent that.

Zip file does not contain _metadata folder. 

I have never encountered this problem when updating through the Google Web Store. That's precisely why I want to clarify that this message only occurs when using the tool and won't affect updates through the Chrome Web Store.

[Oliver Dunk]

Thanks for your patience here - using chrome.runtime.requestUpdateCheck(), I was able to reproduce the issue. Really great catch!

We should definitely file a bug for this so the engineering team can take a look. If you're comfortable doing so, please open one. Alternatively, are you ok with me filing one and uploading your example files as part of it?

Thanks,

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB

[Browser Extenstion]

I would be glad if you could fill out the bug report yourself. I believe you will describe the problem better.
Of course, you can use the extensions I uploaded to demonstrate the issue.

[Oliver Dunk]

Thanks!

I've filed a bug here: https://bugs.chromium.org/p/chromium/issues/detail?id=1492980

Oliver Dunk | DevRel, Chrome Extensions | https://developer.chrome.com/ | London, GB