declarativeNetRequest, excludedRequestDomains and excludedInitiatorDomains Not Working 100%

[Mike-O]

This is about excluding (whitelisting) domains with the declarativeNetRequest API.

I'm finding that even though I do excludedRequestDomains and excludedInitiatorDomains, setting it to an array of something simple like ['x.com'] where x is my test domain like the drudge report website, the redirect rules are still firing regardless. I know this because when I inspect the page, dev tools is showing the redirect. When I set the action.redirect.url to 'about:x', that's what's showing up in the dev tools with a network error.

What's the trick to exclude domains so that when I visit x.com, no rule runs?

[Mike-O]

What's strange is that when I add a listener on chrome.webRequest.onBeforeRedirect in my service worker, and look for details.redirectUrl pointing to about:x, it's not showing me anything. It's only showing the GET about:x net::ERR_FAILED info in the Dev Tools console.

I'm wondering if the web page is trying to pull from its local cache for a resource and then reiterates a previous GET about:x net::ERR_FAILED, and then reattempts it again, and this time, successfully? That would explain the behavior I'm seeing in the browser where the content is blocked when I use the rule without the excludedInitiatorDomains, but the content is not blocked when I use the rule with the excludedInitiatorDomains but still has net::ERR_FAILED errors.

[wOxxOm]

It's either a bug in Chrome or a mistake in your rules.

[Mike-O]

I think I figured it out. You can run your own tests with drudgereport.com, for instance. If the page loads an IFRAME and then inside that IFRAME calls something you are filtering, then excludedInitiatorDomains won't do you any good and the content will still get filtered. That's exactly what's happening. I proved it by going to the page, clicking Network tab in Dev Tools, then checking Preserve Log and Blocked Requests. When I look at the Initiator column, it's anything but drudgereport.com.

I can switch from updateDynamicRules to updateSessionRules and then gain access to excludedTabIds, but the problem there is that I would have to refresh the page once you arrive there in order for me to detect that you browsed to a domain that was whitelisted (no filter), drop rules, reapply rules with your tab ID, then refresh the page. That's not going to work.

I basically want the action to be "Anything on this page needs to not be filtered, no matter what," but the only way to do that is with excludedTabIds as far as I can tell (unless someone shows me otherwise), but then I have the problem of having to intercept (before content loads) that someone surfed to a whitelisted domain, drop rules, get tab id, added it into excludedTabIds, and then refresh the page again, which users will not like because of the page refresh.

[Mike-O]

I tried another technique. I set all my blocking rules to priority 2, and then appended a non-blocking rule with priority 1 like so:

let r = []; // my rules would go here
// (blocking rules code would be here, applied to r using r.push() call, set to priority 2)

// and now add our non-blocking rule, set to priority 1
let o = {};
o.id = ++n;
o.priority = 1;
o.action = {};
o.action.type = 'allowAllRequests';
o.condition = {};
o.condition.initiatorDomains = ['drudgereport.com'];
//o.condition.requestDomains = ['drudgereport.com'];
//o.condition.urlFilter = '||drudgereport.com';
o.condition.resourceTypes = ["main_frame", "sub_frame"];
r.push(o);

I then use chrome.declarativeNetRequest.updateDynamicRules to add this rule and ran my test. Same problem -- if the content I'm blocking is loaded from an IFRAME, then it gets filtered if it matches my filter rules because the initiatorDomain is the domain of the IFRAME, not the domain of the parent.

So, then I tried the requestDomains technique -- same issue.

So, then I tried the urlFilter technique -- same issue.

The only solution I see is the excludedTabIds, but then that means I have to apply rules after arrival at a domain, then refresh the page, which users won't like.

[wOxxOm]

Without seeing your actual extension I can only guess that the source of the problem is that "initiator" in the API is the document that initiates the request, not the main document of the tab. There's no way to filter by the main page URL, which was reported a long time ago in https://crbug.com/1191223 and was ignored since as the team has no one to work on it. I guess you'll have to use chrome.webNavigation or chrome.tabs.onUpdated/onCreated to observe the tabs and dynamically add the rule(s) for tab ids via updateSessionRules.

[Mike-O]

If I add the tab IDs via updateSessionRules, does it act differently and I don't have to refresh the page? I've been using updateDynamicRules for now.

[wOxxOm]

The documentation says tab id filtering is supported only for session rules, which means you don't actually filter by tab id if you use updateDynamicRules.

[Mike-O]

Right, okay, so I switch to updateSessionRules, but then does that mean I have to refresh the page after applying the rules? I hope not -- users won't like that. I noticed with updateDynamicRules that if you change a rule after page loading begins, that it won't run that new rule until you refresh the page. I don't know what happens with updateSessionRules, however. Testing...

[Daniel Marino]

@Mike-O, from quickly skimming your use case, I think an allowAllRequests rule will work for you without having to resort to a tabIds condition. Also, from one of your previous comments I think you might be thinking of priority backward. A higher number priority takes precedent over a lower number priority (I know, this can be counter-intuitive) -- take a look at the example in the DNR documentation where a priority 2 rule gets applied and takes precedent over the priority 1 rule. 

Anyway, if you add a dynamic rule with action allowAllRequests, a urlFilter (or requestDomain -- not initiatorDomain, somewhat confusingly) that matches drudgereport.com, and a larger priority than your filter rules, then it will prevent your lower priority rules from filtering in the main frame and all contained iframes.

For the record, I still think the design I outlined in  https://crbug.com/1191223 makes more sense (is more intuitive) for this use case AND it addresses situations that cannot be expressed in the current DNR implementation. So I would love to get some more folks watching that issue to see if we can move towards writing rules that more directly depend on initiators up the frame hierarchy.