Try out the chrome.declarativeNetRequest API.

[karan...@chromium.org]

Hello developers,

Over the past few months, we have been working on the chrome.declarativeNetRequest API. The API allows extensions to specify declarative rules to block and redirect network requests. It provides a more performant and privacy-friendly alternative to the existing webRequest API since the request is evaluated in the browser itself and it is never exposed to the extension. It’s currently available on the dev channel and will be available on M72 beta.

Please try the new API and let us know of any feedback you have. For any bugs or feature requests, please open a new issue at crbug.com.

Thanks!

[PhistucK]

1. "Array of match patterns which are to removed." - see match patterns.

☆PhistucK

On Mon, Jan 28, 2019 at 4:15 AM Dave Barker <kz...@kzar.co.uk> wrote:

On Tuesday, November 20, 2018 at 10:08:19 PM UTC, karan...@chromium.org wrote:

Please try the new API and let us know of any feedback you have.

I've barely spent much time playing with it, but some things I noticed so far: 

1. I can't see where the format of page_patterns is documented.
   
2. 30,000 is way too low a limit for MAX_NUMBER_OF_RULES. 100,000+ would be more reasonable.
   
3. Element collapsing should remove the element instead of hide it, otherwise you're left with loads of white-space on the page.
   

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To post to this group, send email to chromium-...@chromium.org.
Visit this group at https://groups.google.com/a/chromium.org/group/chromium-extensions/.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/e31ca764-6ff4-4560-9455-72cfe1969145%40chromium.org.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.

[Karan Bhatia]

Thanks for the feedback Dave. Seems like you deleted your message from the thread though.

1. I can't see where the format of page_patterns is documented.
   
2. 30,000 is way too low a limit for MAX_NUMBER_OF_RULES. 100,000+ would be more reasonable.
   
3. Element collapsing should remove the element instead of hide it, otherwise you're left with loads of white-space on the page.

1. It seems like I missed documenting this. Will update the documentation to link to match patterns.

2. This is under review. 

3. I thought this should have worked correctly. Is the screenshot from an extension using declarativeNetRequest? Do you mind filing a bug?

 

You received this message because you are subscribed to a topic in the Google Groups "Chromium Extensions" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/chromium-extensions/qNqURIh4Nss/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chromium-extens...@chromium.org.

To post to this group, send email to chromium-...@chromium.org.
Visit this group at https://groups.google.com/a/chromium.org/group/chromium-extensions/.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CABc02_%2Bpu-osRkX1rKMi9gNUxyMCW%3D9d0XreNxyyOGq5am18oA%40mail.gmail.com.

[Kerry Schwab]

Curious why there's only the ability to add new whitelist entries.

That is, there's no way to add something to a blacklist other than releasing a new version of an extension.

That completely rules out the ability for end users of an extension to add something they want blocked.   Well, other than "suggesting it"  and having it come up in some future release.   Unless we've already hit the 30k entry limit.

This highly narrow ability to change only whitelist entries seems curious.

[karan...@chromium.org]

If you read the API docs carefully, the ability isn't to add allowlist rules, but rather to allow a page/main-frame. This was done so that extensions get the ability to disable their rulesets on certain websites. 

As far as dynamic rules are concerned, we are reviewing whether this is something we want to support.

[Dave Barker]

Karan Bhatia <karan...@chromium.org> writes:

> Thanks for the feedback Dave. Seems like you deleted your message from
> the thread though.

Sure, no problem. I'm not sure why my message was deleted either, it
seemed to post OK, but when I refreshed the page it was gone. I had
assumed I'd replied to the wrong thread or something.

I'll try replying from my email client this time, perhaps that will work.

> 1. It seems like I missed documenting this. Will update the
> documentation to link to match patterns

Ace, thanks. While you're updating the documentation, I noticed a typo
in the first example. The key name there is "resource_types" but I think
it should be "resourceTypes".

> 2. This is under review.

Great, thanks that's good to know. 30,000 is just way too low, take a
look at EasyList alone.

> 3. I thought this should have worked correctly. Is the screenshot from
> an extension using declarativeNetRequest? Do you mind filing a bug?

Yep, that's using the new API. Sure, here's a bug:
https://bugs.chromium.org/p/chromium/issues/detail?id=926136

> As far as dynamic rules are concerned, we are reviewing whether this is
> something we want to support.

I'd urge you to support them, they are really important. For example,
they allow the extension to update its own rules more frequently than
the extension is updated. Another example is that they would allow
"block element" functionality whereby the user selects the element they
would like to block.

I have a question, do the "domains" and "includedDomains" support
wildcards? In other words, is there a way to specify google.com and
all subdomains of google.com? (The Safari API[1] supports that with
"if-domain" and "unless-domain", you just have to prepend "*" to the
domain.)

There are some things I noticed which could make the API easier to use:

4. When reporting problems with rules, it would be useful to provide
more information. For example, instead of "Rule could not be parsed"
it might be nice to say "Unknown key domainss".
5. Also, it would be great if the error message could say the
troublesome rule's ID, instead of its index in the rule list. I
found that confusing. (Often the number's one off, since you have to
start the IDs at > 0.)
6. It would be much easier if the API took care of Unicode characters
in the "urlFilter". It's non trivial for the consumer of the API to
parse a filter in that syntax, split out the domain part, convert
that to punycode, and then deal with converting characters
elsewhere.

Cheers, Dave.

[1] - https://developer.apple.com/documentation/safariservices/creating_a_content_blocker

[Dave Barker]

Dave Barker <kz...@kzar.co.uk> writes:

> There are some things I noticed which could make the API easier to use...

7. Have you considered making isUrlFilterCaseSensitive default to false?
I think that might make more sense since Adblock Plus blocking
filters default to being case insensitive.

Cheers, Dave.

[Kerry Schwab]

I did read it, you're just arguing semantics.

The flow is that you publish an extension with a blacklist.

The api allows you to "whitelist" things in the blacklist, and doesn't allow any other functionality.

That means end users can't "blacklist" something new.   They can only "whitelist" something in the original, static, blacklist.

[karan...@chromium.org]

Ace, thanks. While you're updating the documentation, I noticed a typo 
in the first example. The key name there is "resource_types" but I think 
it should be "resourceTypes". 

Thanks, will update it as well.

Yep, that's using the new API. Sure, here's a bug: 
https://bugs.chromium.org/p/chromium/issues/detail?id=926136 

Thanks for filing, feel free to file bugs and cc me on them!

I have a question, do the "domains" and "includedDomains" support 
wildcards? In other words, is there a way to specify google.com and 
all subdomains of google.com? (The Safari API[1] supports that with 
"if-domain" and "unless-domain", you just have to prepend "*" to the 
domain.) 

I think google.com as part of "domains" should implicilty match maps.google.com without the need for a '*'.

  4. When reporting problems with rules, it would be useful to provide 

    more information. For example, instead of "Rule could not be parsed" 
    it might be nice to say "Unknown key domainss". 

I looked at the code and we do try to be descriptive with error messages. The "Rule could not be parsed" is only shown when the rule is not provided in the correct format. 

 5. Also, it would be great if the error message could say the 
    troublesome rule's ID, instead of its index in the rule list. I 
    found that confusing. (Often the number's one off, since you have to 
    start the IDs at > 0.) 

We chose to use the index since it was always available. For example, sometimes the ID might not be provided. That said, I can see how this can be confusing. 

I'll file a bug for 4 and 5.

 6. It would be much easier if the API took care of Unicode characters 
    in the "urlFilter". It's non trivial for the consumer of the API to 
    parse a filter in that syntax, split out the domain part, convert 
    that to punycode, and then deal with converting characters 
    elsewhere. 

Doesn't the web request api also provide punycode? So I assume the extensions are already handling most of this stuff.

 

7. Have you considered making isUrlFilterCaseSensitive default to false? 
   I think that might make more sense since Adblock Plus blocking 
   filters default to being case insensitive. 

Case sensitive is more efficient and we'll prefer extensions use that. 

[Andrey Meshkov]

Hi all,

I'd like to copy/paste my comments from another thread, it seems that this is a much more relevant place for them.

Here is a short list of what's missing from the declarativeNetRequest. By "missing" I mean that these features are necessary to achieve the same functionality modern ad blockers have with the current webRequest API.

1. Dynamic rules registration.

2. Full regular expressions support. These might be used not so often, but are really crucial in some cases.

3. There is no way to make a blocking rule (or a redirect rule) prevail over an "allow" rule. The thing is that ad blockers use multiple filter lists and it is sometimes necessary to override what's allowed in other filter lists.

4. There is no way to not just block a web request but to prevent opening navigating to a website or opening a new tab ($popup/$popunder modifiers).

5. It is unclear if "redirect" rules allow specifying a "data:" URL or a "chrome-extension:" URL and how it will work with the website's own content security policy. Also, it will make sense to let us create a dictionary of named redirect resources instead of copy/pasting the same redirectUrl (pretty long in case of "data:" URLs) in numerous rules.

6. There is nothing about headers modification there. Although, I don't see how this can be made declarative. Here are some real-life examples: adding CSP headers, modifying "Cookie" or "Set-Cookie" (removing or adding cookies, enforcing same-site policy or modifying max-age), modifying Referer or User-Agent, etc, etc.

7. One more thing that is crucial in some cases is having an option to disable generic blocking/redirect rules (those that have no "domains" key specified). This is important for avoiding ad blocking detection -- a filter list author can disable generic rules for a website and instead specify some more precise rules specifically for that website.

8. Another thing that would be really helpful is having an option to specify origin URL mask and not just the domain name. This is useful for large websites (like google.com for instance) which have different web apps in different locations of the same website (for instance, google.com/maps and google.com/flights).

9. We really need an option to receive feedback on what's done by the declarativeNetApi. What rule was applied to what request? For instance, the webRequest API can be extended to supply this information in one of the callbacks (onBeforeRequest?).

[Andrey Meshkov]

> I think google.com as part of "domains" should implicilty match maps.google.com without the need for a '*'.

The idea is to have "google.*" match all "google.TLD" domains.

[Dave Barker]

On Tuesday, November 20, 2018 at 10:08:19 PM UTC, karan...@chromium.org
wrote:

> Please try the new API and let us know of any feedback you have.

In case it helps anyone try the API out, I've been working on a script
to convert Adblock Plus style filter lists into rules of the format that
the chrome.declarativeNetRequest API expects. Bear in mind it's
experimental!

Here you go: https://github.com/kzar/abp2chromerules

Cheers, Dave.

[Dave Barker]

Thanks for improving the error messages, I'm sure that will make
people's lives easier. I will give your changes a try when I get a
chance. Sorry for the false alarm re element collapsing.

Another idea I had, how about allowing relative redirections? For
example, a redirection to "test.png" instead of
"https://google.com/example/test.png". So far, my script discards
filters which do that, which seems a shame.

> I think google.com as part of "domains" should implicilty match
> maps.google.com without the need for a '*'.

Ah, gotya.

> 30,000 is way too low a limit for MAX_NUMBER_OF_RULES. 100,000+
> would be more reasonable.

I think the output from my script is pretty much correct now, and
FWIW converting EasyList alone gives me approaching 40k rules.

> It would be much easier if the API took care of Unicode characters
> in the "urlFilter".

Scratch this one, turns out we no longer support these unescaped
special characters in filters any more anyway. Also, even if we did,
splitting out the hostname part of a filter is a lot easier than I
first thought.

> Case sensitive is more efficient and we'll prefer extensions use
> that.

Alright, I've done my best to generate case sensitive rules, it's just
tricky since Adblock Plus filters default to case insensitive.

Andrey Meshkov <ay.me...@gmail.com> writes:

> One more thing that is crucial in some cases is having an option to

> disable generic blocking/redirect rules.

I've kind of worked around this one already. My script keeps track of
all the domains with a $genericblock exception, then adding those to the
excludedDomains for all generic blocking filters. But yea, it's not
ideal.

Cheers, Dave.

[Dave Barker]

Karan Bhatia <karan...@chromium.org> writes:

> Please try the new API and let us know of any feedback you have. For
> any bugs or feature requests, please open a new issue at crbug.com.

I've opened another issue[1] for the caret issue Sebastian
raised. Mentioning it here since the issue tracker didn't give me the
chance to copy you in.

[1] - https://bugs.chromium.org/p/chromium/issues/detail?id=932086

Cheers, Dave.

[s.d...@eyeo.com]

Hi,

Cross-posting from the webRequest API thread since this affects declarativeNetRequest API just as much.

Ad Remover, Adblock, AdBlock Plus, AdGuard, Cliqz / Ghostery, MalwareBytes as well Easylist authors (Fanboy and MonztA) have worked together to come up with a shared document, listing out the major issues, the use cases these might affect, as well as technical suggestions on possible ways to improve the APIs and it’s implementation. 

Here is a link to the feedback document. 

We would urge the chromium team as well as the rest of the extensions community to take a look. The recent post from the chrome team on iterating on manifest v3 is encouraging. We hope our feedback provides additional impetus to the changes mentioned by the chrome team in the post, as well as sheds light on what else is needed to improve upon in manifest v3 from the perspective of privacy, ad- and content-blocking extensions.

If the Chrome team needs any details or clarifications on anything mentioned in our feedback, then we would be happy to answer them and discuss more in the forums. 

[Simeon Vincent]

Many thanks to the ad blocking community for this writeup. I'm looking forward to digging into this :)

Simeon

Chrome Developer Advocate

[Thomas Gallagher]

If the declarative net request API is going to be adopted as standard going forward then I can see the following functionality as crucial, if content blocking extensions are not going to get broken:

1) There needs to be the ability to whitelist certain domains so some traffic passes by default. Of course regex is important in some cases, but for me, more important is that the rules database is sensitive to the structure of URLs. Globs can behave unpredictably. If I want to white list "bing.com" but not "climbing.com", the current glob matching in the manifest does not work. So some sort of protocol, host, path, query string matching, via 'contains' or 'equalTo' rules.

2) There must be the ability to add new URLs to the rules database dynamically and quickly at run time. It is often difficult to generalise about traffic so specific URLs may need to be added, and possibly later removed. Bulk additions would also be a useful API.

3) The rules database should have accessible standard attributes of any database, including timestamps for creation, unique IDs, and most importantly developer tags to identify the type of rule, so that bulk deletion operations can be performed.

4) It would be helpful if one could set an expiry date for any given rule at the time of creation, so that the rules database does not end up becoming bloated with collections of out of date rules. I would not be against the rules database insisting on up to date rules in some way.

5) If the rules database could be accessible and modifiable from content scripts, this would actually improve the performance of chrome extensions and would therefore become an improvement to the existing situation, where the messaging API can result in some small delays depending on how busy the extension is.

6) There needs to be some kind of feedback mechanism for a blocking event in the declarative net request API, so that content blockers know when content has been blocked. Again, it would be useful if this event was also available to content scripts. This is crucial for any form of content blocking.

Just a few thoughts.

[Dinesh Bhosale]

Can this API be used to add new headers as well?

For example chrome extension used for changing user agent and testing devices on various sites might require a functionality to add 

a custom user-agent header for certain sites.

Please consider adding a addRequestHeader functionality to ease their work.

[Dinesh Bhosale]

Get requests sent from background page do not have "Origin" header, so they succeed.

Post requests have "Origin" header, so they fail.

We were able to resolve this issue by using 'webRequests' api but it appears it won't be able to remove "Origin" header to make post requests succeed (when manifest version 3 is enforced) then how should a developer implement this functionality to make their extension future proof?

Example:

manifest.json

```

{

  "background": {

    "page": "background.html"

  },

  "browser_action": {

    "default_title": "XHR Test"

  },

  "description": "Cross origin post requests are not working from background pages.",

  "name": "XHR Test",

  "manifest_version": 2,

  "short_name": "TFF",

  "permissions": [

    "*://*.instagram.com/*"

  ],

  "version": "0.1",

  "content_security_policy": "script-src 'self' https://ssl.google-analytics.com; object-src 'self'"

}

```

background.html:

```

<html>

<head>

    <meta charset="utf-8">

    <meta http-equiv="X-UA-Compatible" content="IE=edge">

    <script src="background.js"></script>

</head>

<body>

</body>

</html>

```

background.js:

```

function testGet() {

    var xhr = new XMLHttpRequest();

    xhr.onreadystatechange = function () {

        if (xhr.readyState == XMLHttpRequest.DONE) {

        }

    };

    xhr.open('GET', 'https://www.instagram.com', true);

    xhr.send(null);

}

function testPost() {

    var xhr = new XMLHttpRequest();

    xhr.onreadystatechange = function () {

        if (xhr.readyState == XMLHttpRequest.DONE) {

        }

    };

    xhr.open('POST', 'https://www.instagram.com', true);

    xhr.send(null);

}

testGet();

testPost();

```

Changes were made to content scripts, these changes have forced us to use the background page instead:

https://www.chromium.org/Home/chromium-security/extension-content-script-fetches

But background pages are sending incorrect "Origin" header causing requests to fail, only solution we have at the moment is to use the "webRequest" api but we don't want to use "webRequest" api as it is getting removed in the upcoming versions so how do solve this issue in future proof and backwards compatible way?

We can resolve this issue using "webRequest", "webRequestBlocking", API and manually specifying a custom origin header only for the requests sent by the extension but because "webRequest" API is getting removed how can a developer remove "Origin" header without using "webRequest" api?

Using this code we can solve above issue:

```

chrome.webRequest.onBeforeSendHeaders.addListener(

    function (info) {

        if (info.initiator == 'chrome-extension://' + chrome.runtime.id) {

            for (var index = 0; index < info.requestHeaders.length; index++) {

                if (info.requestHeaders[index].name.toLowerCase() === 'origin') {

                    info.requestHeaders[index].value = "https://www.instagram.com/";

                }

            }

            return {requestHeaders: info.requestHeaders};

        }

    },

    {urls: ['*://*.instagram.com/*']},

    ['blocking', 'requestHeaders']

);

```

Origin header absent on get request causing them to succeed:

Origin header present on post requests, causing them to fail.

Failing request log:

but, when `webRequest` API is deprecated, can `declarativeNetRequest` achieve something similar?

On Wednesday, 21 November 2018 03:38:19 UTC+5:30, karan...@chromium.org wrote:

[Karan Bhatia]

Hi Andrey, 

Extensions Team member here. I was curious about your request for the origin URL mask. 

8. Another thing that would be really helpful is having an option to specify origin URL mask and not just the domain name. This is useful for large websites (like google.com for instance) which have different web apps in different locations of the same website (for instance, google.com/maps and google.com/flights).

We are thinking about adding something similar. Can you elaborate on what your use cases here are? Or how you use this currently in your extension?

Thanks.

[Andrey Meshkov]

Hi Karan,

We had this under development before manifest V3 was introduced. We had to paused it to see how things go.

The proposed syntax is smth like this:
@@||example.org^$domain=example.org&path=/some/pattern/here/*

Here are some cases for this kind of rules:

1. Google Analytics and Google Adwords web apps until they were moved to separate domains. In order to make them work, we had to completely disable filtering there (smth like this @@||google.com/analytics$domain), and we would like to avoid doing this in the future.

2. The same issue with Facebook ad center (facebook.com/*/ad_center) 

3. Different sets of rules for Yandex services. They do different sorts of circumvention so we have to block/unblock different URLs on yandex.ru/news or yandex.ru/maps.

[Karan Bhatia]

Thanks Andrey these cases help. Once we iterate on a proposal, we'll post it publicly for feedback.