Announcement: Improvements to declarativeNetRequest domain matching

[Dave Vandyke]

Greetings fellow extension developers!

We are excited to announce that as of Chrome 101, declarativeNetRequest rule domain conditions are more powerful!

Previously the domains and excludedDomains rule conditions applied only to the domain of the request initiator, and there was no way to match the domain of the request itself other than to use a urlFilter. This behaviour was also a little confusing, since the name “domains” was vague.

To improve on that, we’ve:

1. Renamed the domains and excludedDomains rule conditions to initiatorDomains and excludedInitiatorDomains respectively.

2. Added the requestDomains and excludedRequestDomains rule conditions that match the request domain of requests.

Note: Rules using the old domains and excludedDomains conditions will work for now, but since those rule conditions are deprecated we recommend that you update your rulesets soon!

These changes are available for testing in Chrome Canary, please give them a try and let us know if you find any bugs!

Cheers, Dave.

kz...@chromium.org

https://kzar.co.uk

[wOxxOm]

It'd be nice if you add examples to illustrate the old incorrect approach, the old correct approach, and the new approach.

[Dave Vandyke]

On Monday, March 14, 2022 at 11:05:35 PM UTC wOxxOm wrote:

It'd be nice if you add examples...

Sure, good idea.

So let's say you want to block all image requests made by the example.com website and websites on all example.com subdomains except allowed.example.com. You previously would have had a ruleset like this:

[
  {
    "priority": 1000,
    "condition": {
      "resourceTypes": ["image"],
      "domains": [
        "example.com"
      ],
      "excludedDomains": [
        "allowed.example.com"
      ]
    },
    "action": {
      "type": "block"
    },
    "id": 1
  }
]

That would now look like this:

[
  {
    "priority": 1000,
    "condition": {
      "resourceTypes": ["image"],
      "initiatorDomains": [
        "example.com"
      ],
      "excludedInitiatorDomains": [
        "allowed.example.com"
      ]
    },
    "action": {
      "type": "block"
    },
    "id": 1
  }
]

(Only difference being that domains and excludedDomains are replaced with initiatorDomains and excludedInitiatorDomains.)

The cool thing is, you could now also limit the matches by request domain. Let's say you still want to block all those image requests from the example.com website, but you want to add an exception for images hosted by allowed-image-host.com, you could do this:

[
  {
    "priority": 1000,
    "condition": {
      "resourceTypes": ["image"],
      "initiatorDomains": [
        "example.com"
      ],
      "excludedInitiatorDomains": [
        "allowed.example.com"
      ],
      "excludedRequestDomains": [
        "allowed-image-host.com"
      ]
    },
    "action": {
      "type": "block"
    },
    "id": 1
  }
]

For another example, let's block all images hosted by blocked-image-host.com for all websites:

[
  {
    "priority": 1000,
    "condition": {
      "resourceTypes": ["image"],
      "requestDomains": [
        "blocked-image-host.com"
      ]
    },
    "action": {
      "type": "block"
    },
    "id": 1
  }
]

Hope that helps!

[Deco]

This is quite useful thanks Dave, is this compatible with wildcard modifiers aswell for matching with domains?

Cheers,

Decklin

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/f1750902-a1c6-4070-83e8-c1ee35c77513n%40chromium.org.

[Dave Vandyke]

On Tuesday, March 15, 2022 at 12:19:48 PM UTC decklin...@gmail.com wrote:

This is quite useful thanks Dave, is this compatible with wildcard modifiers aswell for matching with domains?

There's some overlap between the requestDomains/excludedRequestDomains conditions and the urlFilter condition:

• If you want to match the request URL or domain with wildcards, I think you'll need to use urlFilter (or regexFilter).
• For some other use cases (e.g. excluding a request domain), requestDomains/excludedRequestDomains are likely better suited.

It's also perhaps worth noting that a rule can contain both requestDomains/excludedRequestDomains and urlFilter conditions.

Happy blocking :)

[Deco]

Awesome thanks!

[avm99963]

Thanks Dave! :)

[Mike-O]

So, if Chrome version < 101, we can address `excludedDomains`, and >=101, we can address `excludedInitiatorDomains`. Got it.

However, can we add an entry into this array to mean "a domain and all its subdomains"? For instance, can we do `||.example.com` to mean `a.example.com` and `example.com` too? Or, would we do it as `*.example.com`? Or it this not possible at all and we would have to specify each domain and subdomain we want to exclude?

[Dave Vandyke]

> ...can we add an entry into this array to mean "a domain and all its subdomains"?

Yep, in fact that's how domain conditions work with the declarativeNetRequest API. For example, a requestDomains condition of ["example.com"] also matches requests to all subdomains of example.com.

In cases where that is not desired, you can either exclude specific subdomains using excludedRequestDomains/excludedInitiatorDomains, or you can add a urlFilter/regexFilter condition to help restrict matches to the desired subdomains.

Cheers, Dave.

[Mike-O]

Dave,

I think you misunderstand. I'm using the `regexFilter` to filter, but I'm talking about the exclusion domains. Is there a way to wildcard the exclusion initiator domains? The docs don't talk about the wildcard working on the exclusion initiator domain.

[Dave Vandyke]

> Is there a way to wildcard the exclusion initiator domains?

Subdomains are matched (or excluded) in the same way for both requestDomains/initiatorDomains and excludedRequestDomains/excludedInitiatorDomains conditions. So if a rule has an excludedInitiatorDomains condition of ["example.com"], requests initiated by subdomains of example.com would also be excluded.

Cheers, Dave.

P.S. I recently added a testing API declarativeNetRequest.testMatchOutcome that is useful for quickly verifying how declarativeNetRequest rule matching works in practice.

[Mike-O]

Fantastic, thanks. Just what I needed.

[RT]

Hi, this is not working.

Example: https://stackoverflow.com/questions/74391398/mv3-declarativenetrequest-and-x-frame-options-deny?noredirect=1#comment131354998_74391398

Has anyone manage with MV3 for example to put Twitter or Telegram or any other website with "x-frame-options = DENY" working?

On Mv2 it worked perfectly but now this is not working.

Example:

We have a domain "example.com" and there open multiple iframes, for example open an iframe with Twitter.com or Telegram.org. Since twitter.com or telegram.org set the X-Frame-Options to DENY those iframes don't show anything. With MV2 we could run chrome.webRequest and remove those headers and thos iframes load.

With MV3 declarativeNetRequest this is not woking anymore, why?

Greetings.

[RT]

Has anyone made this to work?

I don't understand how such a simple functionality that was running on MV2 stopped working on MV3 with the proper MV3 changes!

Any support here?

[wOxxOm]

You can report the bug on https://crbug.com.

[RT]

To simplify I have made 2 very simple APPS. One in MV2 that is working and the other on MV3 that don't work.

Both open a window that has 2 iframes, one for Twitter and other for Telegram. Both sites use the  X-Frame-Options to DENY.

The MV2 extension removes the X-Frame-Options and works perfectly. The MV3 doesn't. 

How buggy is declarativeNetRequest right now?

Instructions:

Install the MV3 version and test. Then disable/remove the MV3 and install the MV2 and test.

MV3: https://drive.google.com/file/d/1wm_68JvIN8DsattQvSDJpjDzaGgaCqZ-/view?usp=share_link

MV2: https://drive.google.com/file/d/16gb7kZXiCY1PrxR-Cx0zyyjKrAXN9x4Q/view?usp=share_link

MV2 works, MV3 don't work.

Anyone can help?

Greetings.

[wOxxOm]

RT, you need to unregister service worker and clear site data for twitter.com using chrome.browsingData API, this is a known limitation of DNR API, which unfortunately isn't documented. I suggest opening an issue about it.

There's also a mistake in your code: urlFilter doesn't support the same syntax as webRequest, see the documentation for more info. In your case it'd be something like "||twitter.com/" and  "||telegram.org/". 

BTW you can simplify the code: use updateDynamicRules with initiatorDomains + requestDomains inside chrome.runtime.onInstalled listener so it'll run just one time when the extension is installed/updated and the rules will apply to all tabs/windows created by your extension:

  chrome.declarativeNetRequest.updateDynamicRules({
    addRules: [{
      id: 1,
      action: {
        type: 'modifyHeaders',
        responseHeaders: ['X-Frame-Options', 'Frame-Options']
          .map(header => ({ header, operation: 'remove' })),
      },
      condition: {
        initiatorDomains: [chrome.runtime.id],
        requestDomains: ['twitter.com', 'telegram.org'],
        resourceTypes: ['sub_frame'],
      },
    }],
    removeRuleIds: [1],
  });

[Simeon Vincent]

RT, you need to unregister service worker and clear site data for twitter.com using chrome.browsingData API, this is a known limitation of DNR API, which unfortunately isn't documented. I suggest opening an issue about it.

Can you clarify why you need to unregister the service worker and clear site data? I'm not familiar with this "known limitation" and what you described here sounds bad. I did a couple quick searches on crbug.com but nothing stood out.

Simeon - @dotproto

Chrome Extensions DevRel

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/fc60b5ec-b076-48a4-8c27-6f346f0eed41n%40chromium.org.

[Jackie Han]

For "unregister the service worker" (I also don't suggest doing it).

Two years ago, chromium engineer confirmed: " if a SW intercepts a request and returns a cached response, preventing the request from going to the network, the APIs won't see the request."

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAFY0HLM18Vb9Wer5sTH0GPiE6V75u7O5d-QbdKh5Hhuba6M00Q%40mail.gmail.com.

[wOxxOm]

> Can you clarify why you need to unregister the service worker and clear site data? 

https://crbug.com/1175349#c22

[wOxxOm]

RT, regarding my previous comment, the suggestion to simplify the code is not for your use case (I didn't see at first that you add the frames inside another site and not in your extension page). The rest of the comment still holds. I've added an example in https://stackoverflow.com/a/74823389/

[RT]

Hi everyone and specially wOxxOm,

on a preliminary tests I can confirm to everyone that wOxxOm claims are true! 

For telegram.org the X-Frame-Options headers are removed and iframe is loaded, but for twitter.com they are not. I had to add other permission: "browsingData" and then run this piece of code:

chrome.browsingData.remove({
  "origins": ["https://twitter.com"]
}, {
  "serviceWorkers": true
}, callback);

Only after running that piece of code that twitter.com X-Frame-Options headers are removed and twitter.com is successfully run on the iframe.

I have wasted dozens of hours to discover this bug on declarativeNetRequest that only wOxxOm was able to find. This is undocumented and anyone out there will find this issue very quickly. I have used twitter.com to find this bug but I can confirm that there are thousand of other sites that have the same behavior as twitter.com and the declarativeNetRequest is buggy.

In this case I had to remove X-Frame-Options header, but I would assume anyone that want to do different things using declarativeNetRequest they will not work properly on some websites that have some service-worker configuration that causes declarativeNetRequest to not work

wOxxOm I also confirm what you say in your last comment, on this specific case the  declarativeNetRequest.updateDynamicRules don't work, but I have seen your post on stackoverflow and I will test it out, but in a quick look it looks like it will work that way.

Anyway I will post here and on stackoverflow when I reach final testing and conclusions.

I also have to say that MV3 has been a set back. I had to removed some functionality that was present on MV2. Example: load twitter on backgroundpage and monitor new messages, notifications and show them on the Extension badge without twitter open on a webpage. This is impossible on MV3.

I have also noticed some random bugs on MV3 like alarms don't fire or communication between different extensions don't wake up the other extension. Example: send message from extension A to Extension B and extension B the service worker don't wake up.

I don't understand why Android on new versions get more powerful and more features. But on the extensions it's the oposite. Same company different views and approaches.

Greetings.

[wOxxOm]

>  load twitter on backgroundpage and monitor new messages

Use chrome.offscreen API. Currently it's still in development, so you'll need Chrome Canary (Chrome Dev channel might work as well).

[RT]

Hi  wOxxOm and everyone,

I have made further testing and at least on another site declarativeNetRequest is not working. For example opening web.whatsapp.com it shows a "Sorry, something went wrong." and on the console on the Network shows and HTTP 400.

I have updated the code to the wOxxOm suggestions and include web.whatsapp.com here:

https://drive.google.com/file/d/1Cck_AXXGOA83PZmU0Hsw_NTD7h70NP6x/view?usp=sharing

Any help?

Greetings.

[wOxxOm]

Assuming this site doesn't work in MV2 webRequest as well, it's probably because it checks `window == top` in JS. The only way to remove the check is to use chrome.debugger API with Fetch commands.

[Dave Vandyke]

On Thursday, December 15, 2022 at 2:24:05 AM UTC Simeon Vincent wrote:

Can you clarify why you need to unregister the service worker and clear site data? I'm not familiar with this "known limitation" and what you described here sounds bad. I did a couple quick searches on crbug.com but nothing stood out.

I think Jackie and w0xx0m have already answered you, but in case it helps I have created a test page + extension that demonstrates the problem. While chrome.webNavigation events fire for ServiceWorker cached (main-frame) requests, chrome.webRequest events do not and declarativeNetRequest rules cannot match them either. I figure this could be a way for websites to circumvent ad blocking and similar extensions.

[Efraim Bart]

> 1. Renamed the domains and excludedDomains rule conditions to initiatorDomains and excludedInitiatorDomains respectively.

How does one add no initiator domain to one of these lists? If I want to include or exclude any requests initiated by the plugin I can add `chrome.runtime.id` to the above properties, but what about a request initiated by the user (e.g. opening a new tab and entering the url directly into the address bar)?

[wOxxOm]

It depends on your rule's condition. If your other conditions like urlFilter are already sufficient then you don't have to specify initiatorDomains. You can specify it if the other conditions are for a sub-resource of this site or when you want to make the rule faster by limiting the sites where its other conditions will be checked, which is especially helpful with a slow regexFilter.

[wOxxOm]

...and in such cases it'll be the domain of the site.

[Efraim Bart]

I don't understand what you are suggesting. No other condition can limit the rule to only apply if there is an initiator domain.

[wOxxOm]

Seems I don't understand you now or maybe I didn't since the very beginning. Maybe you could describe your use case in more detail.

[wOxxOm]

Found your post on stackoverflow, commented there.