declarativeNetRequest Rule Priorities

[Theodore Dubois]

Hello developers,

We're exploring adding support for generic priorities to the
declarativeNetRequest API. Currently, rule priorities are mostly
fixed: allow rules are higher priority than block rules; block rules
are higher priority than redirect rules and redirect rules support the
“priority” key that determines their relative priority. This change
would allow the priority of all rules to be specified as an integer,
giving the extension greater control over the order of evaluation.
Dynamic rules will still always override static rules.

We've also heard requests from developers for a rule action that skips
evaluation of a set of rules. For example, there could be a dynamic
rule that specifies that other dynamic rules should be ignored but
static rules should still be matched. Let us know if that's something
that might be useful for your extension as well.

Thanks,
~Theodore

[michael Bostwick]

Hi,
     Thanks for sharing. Instead of generic priorities where everyone aims for a higher number, I'd suggest tags or intents. Being abstract around the targets means people will just aim to take the higher numbers, if you use intents it would be easier to isolate or even tune how intents should be applied, similar to the way settings overrides get applied.  You could also add permissions to the manifests to better scope and address the rules.

Thanks,
Michael Bostwick

[Andrey Meshkov]

Hi Theodore,

This is awesome!

I guess these priorities should be considered within a single extension only.

If some extension decides to block a request, the request must stay blocked regardless of other extensions' rules.

> Dynamic rules will still always override static rules.

Why? This will cause a lot of confusion.

Consider the following example.

1. There are filter list maintainers that use a content blocking extension to develop their filter list and test new rules.

2. These new rules are added as dynamic rules and override any static rules.

3. At some point, the content blocking extension is updated, and these new rules make it to the static ruleset.

4. The rules' behavior is different now, and they don't do what they were doing before.

> We've also heard requests from developers for a rule action that skips evaluation of a set of rules.

Not sure about skipping "a set of rules", but I am sure it'd be really useful to be able to somehow disable a single target rule.

For instance, there could be a problematic rule in the static rules set, which redirects too or unblocks too much.

In the current implementation, there's no simple way to deal with it with dynamic rules, your only option is to change the static rules set.

[Theodore Dubois]

We're planning to avoid this problem by not comparing priorities from different extensions. This will likely end up working similar to blocking webRequest, so if one extension blocks a request, that will take priority over another extension allowing the request.

~Theodore

[Karan Bhatia]

We're planning to avoid this problem by not comparing priorities from different extensions. This will likely end up working similar to blocking webRequest, so if one extension blocks a request, that will take priority over another extension allowing the request.

Elaborating (and correcting) this slightly. The priority system is only relevant for a single extension. When multiple declarativeNetRequest extensions are installed, the least recently installed extension gets priority. This is how webRequestBlocking and declarativeNetRequest work today.

[Karan Bhatia]

1. There are filter list maintainers that use a content blocking extension to develop their filter list and test new rules.

2. These new rules are added as dynamic rules and override any static rules.

3. At some point, the content blocking extension is updated, and these new rules make it to the static ruleset.

4. The rules' behavior is different now, and they don't do what they were doing before.

Not sure I get this. Why does the rule behavior change at step 4. Also, won't the extension delete the relevant dynamic rules after step 3? 

[Andrey Meshkov]

Hi Karan,

> Not sure I get this. Why does the rule behavior change at step 4. Also, won't the extension delete the relevant dynamic rules after step 3? 

I am referring to the point that dynamic rules always have greater priority than static rules:

> Dynamic rules will still always override static rules.

So when you test new rules in the dynamic ruleset, their behavior is different from the case when these new rules are moved to the static ruleset.

[Karan Bhatia]

Hi Andrey, 

Let me try to elaborate on how I am thinking about dynamic rules.

I see two use cases of dynamic rules:

1. Rules added in response to user action. Say disable the extension on example.com, block all third party scripts etc.

2. Rules pushed to extensions to augment their static rulesets.

Now it's natural to think that rules in 1. should generally always be higher priority than other rules since they were added in response to user action.

For rules added for use case 2., like you said, the way it might work is:

1. New rules are pushed to extensions to augment the static rules. 

2. These are added as dynamic rules and override any static rules. 

3. The extension is updated, the new rules make it to the static ruleset. Priority of rules within the new published static ruleset is arranged appropriately by the extension devs. (The new rules can still have higher priority than the particular static rules they were supposed to override).

4. Extension deletes the new rules from the set of dynamic rules, so that the only set of dynamic rules remaining are the ones added in response to user action.

It's not clear to me how things break here.

The problems I see with static and dynamic rules sharing the same priority space is:

• Extensions would need to be careful that dynamic rules added in response to user action don't change behavior once the extension is updated. In practice, this would mean the extension maintaining a fixed upper limit of static rule priority across extension updates and a fixed minimum priority for actions added in response to user action. This is easy to get wrong.
• Static rules will have immutable priorities. Adding new dynamic rules to the same priority space would mean additional book-keeping on the part of extensions. 
• It's not immediately obvious to me if this solves any additional use case. Let me know if you can think of any.

Not sure about skipping "a set of rules", but I am sure it'd be really useful to be able to somehow disable a single target rule.

For instance, there could be a problematic rule in the static rules set, which redirects too or unblocks too much.

In the current implementation, there's no simple way to deal with it with dynamic rules, your only option is to change the static rules set.

Can't the extension add an appropriate set of dynamic rules to "shadow" the problematic static rule. Granted it might not be ideal.

[Andrey Meshkov]

Hi Karan,

> Priority of rules within the new published static ruleset is arranged appropriately by the extension devs. (The new rules can still have higher priority than the particular static rules they were supposed to override).

There's an important point missing here -- extensions' developers do not maintain filter lists.

Filter list maintainers are the ones who should decide what priority the rule has.

What if "the user" from the Point 1 is not a regular user, but an Easylist maintainer who is testing new rules? 

Due to its size, Easylist most likely will be baked into extension's static ruleset, and we'll run into the described issue: 

once the new rule makes it to the static ruleset, it's behavior changes from when the list maintainer was testing it.

I'll try to come up with a detailed explanation with some examples by Monday.

> Can't the extension add an appropriate set of dynamic rules to "shadow" the problematic static rule. Granted it might not be ideal.

It would be extremely hard to shadow a problematic "whitelist" rule.

[Andrey Meshkov]

Hi Karan, sorry for the late reply!

Let's get back to this issue.

> Now it's natural to think that rules in 1. should generally always be higher priority than other rules since they were added in response to user action.

It is actually not that obvious to me. First of all, I am talking about the case when the user is a filter list maintainer, they need to be able to see how the list will behave when it's part of the static ruleset.

Second, adding a rule in the response to a user action is not the only use case for dynamic rules.

Filter lists differential updates is another case, and in some configurations, newly added rules should not override existing rules.

If you introduce the term "filter list" to static/dynamic rulesets (and I think you should), it is common for different filter lists to negate each other when it's necessary.

For instance, a "French" filter list may disable problematic "Easylist" rules on some french website, but this can be broken when a differential update is added via dynamic rules.

> Extensions would need to be careful that dynamic rules added in response to user action don't change behavior once the extension is updated. In practice, this would mean the extension maintaining a fixed upper limit of static rule priority across extension updates and a fixed minimum priority for actions added in response to user action. This is easy to get wrong.

Tbh, I don't see how can this be possibly got wrong.

> Can't the extension add an appropriate set of dynamic rules to "shadow" the problematic static rule. Granted it might not be ideal.

I've just realized that there's another really important use case for the "skipping" functionality. I am talking about differential filter list updates. Currently, we can add new rules via dynamic rules, but there's no simple way to skip or disable rules that are no more in the filter list.

PS: Could you please also take a look at this post? 

https://groups.google.com/a/chromium.org/forum/#!topic/chromium-extensions/CA4Bbh4TSi8

[Karan Bhatia]

Hi Andrey, I have replied on your other post. Let me know if you have any feedback.