DevTools protocol interception, blocking & modification of network requests

[Alex Clarke]

A common request for headless chrome has been for the ability to block or modify network requests via DevTools protocol.  Assuming the patch stays landed, Chromium (as of 61.0.3119.0) now has the ability to do this via a few new Network domain command and events.  

You can use this to:

• Block resources, e.g based on url or request headers
• Modify the request, perhaps by changing the url, or the request headers, or the HTTP method (e.g. change a GET into a POST)
• Mock the server response by providing the raw bytes of the response (HTTP status line, headers etc...)
• Observe and modify redirects

Node style pseduocode to illustrate some of things that are now possible:

await Network.Enable();
await Network.enableRequestInterception({enabled: true});
Network.requestIntercepted((params) => {
  let continueParams = {interceptionId: params.InterceptionId};
  if (params.request.url.endsWith('.jpg')) {
    // Pretend the .jpg IP address was unreachable.
    continueParams.errorReason = 'AddressUnreachable';
  } else if (params.hasOwnProperty('redirectStatusCode') &&
             params.redirectStatusCode == 302) {
    // Pretend the server sent a 404 instead of a 302.
    continueParams.rawResponse =
        btoa("HTTP/1.1 404 Not Found\r\n\r\n");
  } else {
    // Allow the request to continue as normal.
  }
  Network.continueInterceptedRequest(continueParams);
});

Page.navigate({url: 'http://some-website.tld/'})

To view release status, see: https://storage.googleapis.com/chromium-find-releases-static/6f5.html#6f5b3053ff11488b985c18e8d03472356e049fd5

[Boris Okunskiy]

Hi Alex,

That's amazing news! Do you think it would be possible to handle proxy authentication this way? Perhaps the second "most wanted" feature from my list is to be able to manage proxies via CDP in a same way it is already possible via Chrome Extensions.

[Alex Clarke]

That's something we could do, please star https://bugs.chromium.org/p/chromium/issues/detail?id=729525 if you think that's functionality you would use.

--
You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/cf2b1738-4aa5-4199-92fc-0c825af1de6f%40chromium.org.

[excheq...@gmail.com]

Alex,

Thank you for this. I had been watching this particular item very closely ever since the proposal document on it came out, and I'm super stoked that it has finally landed!

One thing I've noticed is that when Chrome is started with the --proxy-server flag, interception doesn't work. Meaning, Network.requestIntercepted doesn't fire.

Should I file a bug for this?

[Alex Clarke]

That sounds like a bug. I'm not exactly sure how --proxy-server works under the hood, perhaps when set a different URLRequestContext is used.

I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=732499 to track this.

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/52b30bb1-83b2-4097-9c62-9b7c1c2e2362%40chromium.org.

[Alex Clarke]

Implemented by https://chromium-review.googlesource.com/c/535695/3.  Should be in the canary build tomorrow.

On 5 June 2017 at 09:06, Boris Okunskiy <bo...@ub.io> wrote:

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/cf2b1738-4aa5-4199-92fc-0c825af1de6f%40chromium.org.

[Alex Clarke]

Heads up for any one using DevTools request interception, Network.enableRequestInterception got renamed to Network.setRequestInterceptionEnabled in this patch https://chromium-review.googlesource.com/c/563385/  That landed in 61.0.3152.0.

[Boris Okunskiy]

Hey guys,

I was trying to use this new awesome `Network.setRequestInterceptionEnabled` — but found out it is only enabled in `--headless` mode.

Is there any way/plans on making it available in regular versions as well (maybe via a separate cli flag?)

Thanks in advance, keep up the good work!

— Boris

On Monday, June 5, 2017 at 10:38:29 AM UTC+3, Alex Clarke wrote:

[Alex Clarke]

On 1 August 2017 at 11:51, Boris Okunskiy <bo...@ub.io> wrote:

Hey guys,

I was trying to use this new awesome `Network.setRequestInterceptionEnabled` — but found out it is only enabled in `--headless` mode.

Is there any way/plans on making it available in regular versions as well (maybe via a separate cli flag?)

It should work on all versions of chrome.

 

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/76ece097-9393-49dd-a3c9-ccfb1e5cfc00%40chromium.org.

[Alex Clarke]

Caveat all recent versions :)

[Boris Okunskiy]

Oh, then I think it might be a bug after all: removing the `--headless` flag causes `Network.requestIntercepted` events to stop firing at all.

I'm on Chrome 62.0.3173.0 (Official Build) canary (64-bit) (Mac OS X)

Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

To post to this group, send email to headle...@chromium.org.

[Alex Clarke]

It should defiantly be in 62.  Can I just confirm you're not running this in vm or a container?

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/3227bd22-eb90-4d9f-9e8e-0ee08109cdfb%40chromium.org.

[Boris Okunskiy]

Running locally on Mac, and it works perfectly with `--headless` flag. Strange, it also appears to work in normal mode after I remove the `--user-data-dir` flag (which points to a non-existent tmp dir).

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/3227bd22-eb90-4d9f-9e8e-0ee08109cdfb%40chromium.org.

[Alex Clarke]

Ah that's interesting. I wonder if there's something unusual about your StoragePartiton in that case. I'll have a look.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/e449d385-a491-4587-b93a-5894008d0697%40chromium.org.

[Boris Okunskiy]

Thank you for taking a look!

Among other things I found that pre-creating a user directory has no effect. Permissions, names, etc also do not seem relevant here. I'm happy to provide more details to help with investigation.

Thank you!

— Boris

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/e449d385-a491-4587-b93a-5894008d0697%40chromium.org.

[Alex Clarke]

I tried to reproduce this on Linux and it's working with or without --user-data-dir (I tried valid existing, valid non-existing and non-existing paths).

Are you able to try your setup on another platform?

Incidentally the API changed a few weeks ago so here's a current working example of using this stuff:

var exampleSocket = new WebSocket("ws://localhost:8080/devtools/page/HASH");  // Change the websocket url as needed to match your setup

exampleSocket.onopen = function (event) {

exampleSocket.send('{"id": 1, "method": "Network.enable", "params": {}}');

exampleSocket.send('{"id": 2, "method": "Network.setRequestInterceptionEnabled", "params": { "enabled": true}}');

};

var id = 3;

exampleSocket.onmessage = function(event) {

    console.log(event.data);

  var message = JSON.parse(event.data);

  console.log(event.data);

  

  if (message.method == "Network.requestIntercepted") {

    this.send(JSON.stringify({"id": id++, "method": "Network.continueInterceptedRequest", "params": {"interceptionId":  message.params.interceptionId}}));

    return;

  }

};

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/6138d1f6-9efb-4562-94bb-d0aed8bcb7d0%40chromium.org.

[Boris Okunskiy]

Thank you, Alex!

I just tried running my code on Linux (Chromium 62.0.3176.0), and it indeed works as expected. So it seems like the `Network.requestIntercepted` event doesn't fire only on Mac OS X, in non-headless mode, and only when including --user-data-dir.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/6138d1f6-9efb-4562-94bb-d0aed8bcb7d0%40chromium.org.

[Alex Clarke]

Thanks for the report, I've filed https://bugs.chromium.org/p/chromium/issues/detail?id=752010 to track this issue.  I'll see if I can reproduce this on a mac.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.

To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/2f5248eb-e2ca-4210-8175-b023852a8142%40chromium.org.

[seba...@madumbo.com]

Hi, 

Is it possible to use this feature for having an aggressive cache? My use case is to be able to "reload" a page very efficiently by caching all the initial requests it made. Basically: 

- load a page 

- intercept and store all the requests

- do some automated testings that require at some point to reload the page (a.k.a. "resetting" the page to the initial state)

- reload the page fast by intercepting all the reloading requests and serving same the same previous content to avoid (almost) any network calls

My problem is that Network.requestIntercepted requires a rawResponse which is the full response (with status line, headers and body) but I'm only able to get the first response bodies using

Network.responseReceived((response) => {
  const body = client.Network.getResponseBody({requestId: response.requestId})
  // cache the body
})

But reconstructing the full response has been unsuccessful so far (I tried take the base64 encoding into account, and adding the response headers and status).

So my questions are:

- is there some other way I missed to get the full response of a request?

- can my method work or I am completely wrong? If so, would you see another strategy to "reset" the page to the initial loading in an efficient way (browser caching is not enough, I need to avoid network calls)?

Sorry for the long question, and thanks in advance!

[Alex Clarke]

On 12 October 2017 at 09:37, <seba...@madumbo.com> wrote:

Hi, 

Is it possible to use this feature for having an aggressive cache? My use case is to be able to "reload" a page very efficiently by caching all the initial requests it made. Basically: 

- load a page 

- intercept and store all the requests

- do some automated testings that require at some point to reload the page (a.k.a. "resetting" the page to the initial state)

- reload the page fast by intercepting all the reloading requests and serving same the same previous content to avoid (almost) any network calls

It ought to be possible to do this.  That said it may or may not be faster because there is overhead in interception and in passing the base64 encoded resources in.  Of course if your network is slow this will probably be a win.

 

 

My problem is that Network.requestIntercepted requires a rawResponse which is the full response (with status line, headers and body) but I'm only able to get the first response bodies using

Network.responseReceived((response) => {
  const body = client.Network.getResponseBody({requestId: response.requestId})
  // cache the body
})

But reconstructing the full response has been unsuccessful so far (I tried take the base64 encoding into account, and adding the response headers and status).

In theory that should work, although the response headers and HTTP status line are important.  You should be able to get those from Network domain notifications.

 

So my questions are:

- is there some other way I missed to get the full response of a request?

A proxy might help.

 

- can my method work or I am completely wrong? If so, would you see another strategy to "reset" the page to the initial loading in an efficient way (browser caching is not enough, I need to avoid network calls)?

In theory this ought to work but the surest way of doing this is writing your own proxy.

 

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/02b93892-baf6-48b5-a54e-711fee279e5b%40chromium.org.

[seba...@madumbo.com]

Thanks for your insight. I want to avoid writing an external proxy so I'll give this solution an harder look instead, now that you confirm it should theoretically work, but I'll keep in mind the overhead.

Thanks !

Le lundi 5 juin 2017 09:38:29 UTC+2, Alex Clarke a écrit :

[pixel...@gmail.com]

I've been experimenting with this feature, and its been quite helpful in place of a proxy server. I've noticed unusual behavior when base64 encoding certain content for the rawResponse field, in particular if there's unicode data. For example, the page could render completely wrong, or unusual characters appear. The MDN docs for btoa() specifically calls out that certain types of data--like unicode--need special attention when encoding/decoding.

Is there any guidance for how to handle encoding this type of data?

[Alex Clarke]

Do you have any example characters which render incorrectly?

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/3e3576ed-6290-4671-81d9-c05e9699627b%40chromium.org.

[excheq...@gmail.com]

I was curious about utf-8 encoding in btoa(), so I tried this - 

atob(btoa("example ❤️🙂🌎"));

It throws an error - Uncaught DOMException: Failed to execute 'btoa' on 'Window': The string to be encoded contains characters outside of the Latin1 range.

The solution suggested in MDN works - https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding#The_Unicode_Problem

b64DecodeUnicode(b64EncodeUnicode("example ❤️🙂🌎"));

returns

sdasd❤️🙂🌎

On Monday, June 5, 2017 at 1:08:29 PM UTC+5:30, Alex Clarke wrote:

[Alex Clarke]

Right that makes sense.  As long as the unicode characters are encoded correctly they should work in the rawResponse.  At that level it's just a byte stream.

--

You received this message because you are subscribed to the Google Groups "headless-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev+unsubscribe@chromium.org.
To post to this group, send email to headle...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/headless-dev/fc261736-8da0-44c6-8424-67f589de0eb9%40chromium.org.

[pixel...@gmail.com]

Right, the MDN approach works, but if you were to encode the rawResponse field using the MDN b64EncodeUnicode function, will Chrome decode the response value the same way as MDN's b64DecodeUnicode? If it doesn't you won't get the same value out. I would the value is not decoded in this fashion, but I have not tested it. I'll try this out some this evening using the node CDP package (my work thus far has been in Go using github.com/wirepair/gcd) and see what happens.

To unsubscribe from this group and stop receiving emails from it, send an email to headless-dev...@chromium.org.

To post to this group, send email to headle...@chromium.org.

[pixel...@gmail.com]

Sorry for the slow response. I've done some more experimenting, and the weird behavior I've seen seems to be due to the quirks of the HTML parser I'm using. From what I've been able to gather so far the base64 encoding is working as intended, even when I've included unicode characters.