Monitor web requests of a tab with an extension and use tab's console to log results

[Bogdan Nazaruk]

Hello hello!
My progress with my debugging extension has been quite spectacular with my extension and your help so far. Now it's time to implement another piece of functionality. And I'd like an advice and a sanity check.

Again, manifest v3.

This time, I want to add a listener to all network requests of a given active tab. For it, I'm thinking to use this: https://developer.chrome.com/docs/extensions/reference/webRequest/ either in the content script or the backround script. Probably background. I want it to work without the popup's involvement whatsoever. The popup will only have a config option to disable or enable this behavior for every site, any active tab.

So Background would have to check if debugging is enabled in the extension settings (I plan to store the settings in the extension's local storage). Then it deploys the listener to the network requests. And here I encounter my first caveat. You see, from that request I need two things: The full url of the request and the response headers. That means I can't just use the onBeforeRequest. I need to use something like onHeadersReceived. But I'm not sure if onHeadersReceived would have the original request url in its data. onResponseStarted seems like it has the response code. onCompleted should work too, but I don't really need to wait for it to complete. Once I know the response (and the request url), I'm good.

Any advice on which callback/listener I should use?

Background would then:
1. Check if the extension's debugging is on;
2. Check each response, and collect responses, request(!) urls of which match the mask
3. Check response codes and get rid of all that's not 200
4. Send a window message toooo... Well, to this tab's injected script. I hope I can avoid content script here and just directly send it to my message listener within the tab.

That injected script's listener will parse the message it gets from background and neatly log it in the console with a lot of cool bells and whistles, of course.

Thanks!

[wOxxOm]

Indeed, `url` in onHeadersReceived is the response's URL, which may be different if the request was redirected. You can listen to both events and store the initial URL in a map:

const urls = new Map();

chrome.webRequest.onBeforeRequest.addListener(info => {

  urls.set(info.requestId, info.url);

}, config, extras);

chrome.webRequest.onHeadersReceived.addListener(info => {

  const reqUrl = urls.get(info.requestId);

  urls.delete(info.requestId);

  .............

}, config, extras);

chrome.webRequest.onErrorOccurred.addListener(info => {

  urls.delete(info.requestId);

}, config, extras);

> either in the content script or the backround script

Content scripts can't use most of `chrome` API, so the background script it is.

> avoid content script here and just directly send it to my message listener within the tab

Assuming you mean the script in the MAIN world, you can do it via externally_connectable messaging. I suggest using chrome.runtime.connect in that script + onConnectExternal in the background script because there may be a lot of messages per second. onConnectExternal would store the incoming `port` in a global map with a key on `documentId` which is present inside onConnectExternal's `sender` parameter and webRequest's info parameter. Use onDisconnect even of the port to delete it from the map.

[Bogdan Nazaruk]

Thanks a lot! I've tried this approach, but ran into a few issues.

After glimpsing at the chrome runtime connect, I can see that it does a lot more than I need. I don't like how it seems to be sending events to all tabs. Plus I don't expect too many messages to be passed back and forth. It's just one endpoint that I'm monitoring, essentially. Plus, it looks like I'm able to log into window's console from my content-script, so I'm fine with just passing events from background to content-script. I tried to do simple await chrome.tabs.sendMessage(tab.id, msg) but I'm having troubles.

Another approach I had in mind was doing it all via the injected dom script. I could monitor all network requests through the browser's performance api. But It would be dirty. I would have to implement some polling to analyze them every second or so, not having listeners. Anyhow, a few questions:

1. The first issue is that I don't understand when the service worker is being triggered. Is it just triggered once on extension initialization and never ran again?

2. Do we have sensible tools to debug the service workers? Insert breakpoints and inspect the state?

3. I'm trying to deploy listeners just like you've suggested, but it looks like they have odd requirements. For example, the filter attribute of the listeners is mandatory. The url field in it is mandatory too. Can't skip it. And that's not a problem. The documentation gives this example for the filter: https://developer.chrome.com/docs/extensions/reference/webRequest/#type-RequestFilter and then it links to this page for general rules on how to build url patterns: https://developer.chrome.com/docs/extensions/mv3/match_patterns/ I spent an hour trying to persuade it that my filter is a valid one, but it disagrees. I'm only interested in network requests sent to any url that contains /b/ss/ in the path. I would much prefer to just use a regexp for that, but it insists on me using the matching pattern system, so the best thing I came up with was this:

  const filter = {

    urls: ["*/b/ss/*"],

    tabId: tab.id

  }

Aaand it still says wrong filter. In particular, it says: Unchecked runtime.lastError: '*/b/ss/*' is not a valid URL pattern.

Suggestions would be appreciated.

4. I'm failing to pass any message to the content script to just test if the message passing works. Because if I just pass a message from the background script to the content script, I get an error saying the listener doesn't exist. Which kinda makes sense. I guess background resolves before content-script so it can't find a content script in an active tab to which I'm trying to send the message. 

So I added a try catch to avoid crashing my background script due to the error...

async function sendURLToCS(tab, url){

  try{  

    const response = await chrome.tabs.sendMessage(tab.id, {greeting: "hello", url: url});

  } catch(e){

    console.log("Error: " + e);

  }

}

async function test(){

  console.log("check if I can read this in the service workers log...")

  const timeoutId = setTimeout(async () => {

    try{

      const [tab] = await chrome.tabs.query({active: true, lastFocusedWindow: true});

      sendURLToCS(tab, "TESTING BACKGROUND!");

    } catch(e){

      console.log("Error: " + e);

    }

  }, 1000);

}

test();

Then in CS I just have a copypaste from the documentation: 

chrome.runtime.onMessage.addListener(

  function(request, sender, sendResponse) {

    console.log(sender.tab ?

                "from a content script:" + sender.tab.url :

                "from the extension");

    if (request.greeting === "hello")

      sendResponse({farewell: "goodbye"});

  }

);

No logging from the callback. So looks like that onmessage is never triggered. 

Any suggestions here?

Thanks!

[wOxxOm]

1. connect in the injected script connects to any chrome-extension:// context that has onConnectExternal, so the service worker will be the only recepient. It's the only way to communicate directly with the MAIN-world script ("injected script"). For the sake of completeness there is also a much much more complicated method of temporarily adding a web_accessible_resources iframe and transfering a MessageChannel port that would allow passing of binary arrays and blobs between the service worker and the injected script.
   
   
2. service worker wakes up automatically when you define a `chrome` event listener e.g. chrome.runtime.onConnectExternal
   
   
3. Use devtools to debug the service worker: open chrome://extensions page, enable "developer mode" switch at the top, then click "service worker" link in your extension's card.
   
   
4. A url pattern must have a scheme+host part, so your pattern should look like "*://*/*/b/ss/*" and to observe all urls use "<all_urls>"
   
   
5. Chrome doesn't run content scripts in the existing tabs after you install or reload the extension in chrome://extensions page, so you'll have to reinject them explicitly. However, it's better to use `connect` in the injected script directly so you won't even need the content script at all, at least if the only thing it does is relaying of messages to the MAIN-world script and injection of the latter, which can be performed directly in the service worker's chrome.runtime.onInstalled event listener using chrome.scripting.registerContentScripts with world: 'MAIN' parameter.

[Bogdan Nazaruk]

1. You talk about the injected script, but I don't need to talk to it. After you showed the chrome.scripting.executeScript I implemented all through it and it worked like a charm. I had to stringify some objects cuz they wouldn't fit it, but it all worked well in the end. I've got all I needed. Now I'm good with content-script doing console logging and talking to the background worker. I think I'll remove the injected script completely. No need in it anymore, I think.

4. Wow! That is an awkward design. Even after reading the page on patterns, I couldn't digest this. Thanks! It stopped throwing errors!

5. I noticed. I just reload the page and it repopulates. It's not a problem. I don't need to communicate with the main world. I think main world is the world that has the tab's window's JS context. So like content-script is abstracted from there, having only the DOM and the access to write into the Main world's console. That's really all I need. Don't want to overengineer without the need.

6. A similar functionality to what I'm trying to achieve is already implemented in a different extension, but that one is not supported, buggy and manifest v2. I still glimpsed into it. Here it is: https://chrome.google.com/webstore/detail/debugger-for-adobe-analyt/bdingoflfadhnjohjaplginnpjeclmof Look at its background script. It's incredibly simple. All it needs to do is chrome.tabs.sendMessage(details.tabId, details); And poof! the request is sent. And then in cs they just chrome.extension.onMessage.addListener(...) And poof! Message received! It also communicates with the page script to grab a few more details, but I'm fine not doing that for now. Did manifest v3 remove that communicational simplicity? From the documentation it looks simple enough.

[wOxxOm]

Your latest message says you're using executeScript, which is fine, but it wastes time to serialize and then execute the function, assuming it's what you use, so a better approach is messaging. 

Your initial message said you didn't want the content script and that you wanted to communicate with the injected script, so I guessed you meant a MAIN-world script. Re-reading your messages, I see "content script" and "injected script".

• An injected script is an old wide-spread term for a script in the MAIN world, executed via DOM element in ManifestV2 or world:'MAIN' in ManifestV3. Theoretically it could also mean any content script, but it's rare, even though it's logical since any such scripts are indeed injected.
• A content script in ManifestV2 meant the script declared in content_scripts or injected via executeScript. In ManifestV3 in addition to these two a very unwise decision was made by chrome extensions team to extend the name content script to the scripts in the MAIN world, even though they don't have any of the powers/properties typically associated with a content script. Now the term content script doesn't have an inherent meaning that it used to have for 10+ years and we have to say "MAIN-world script" and "content script in the default world".

So, assuming your extension activates only on demand, the most efficient approach would be to run the content script just once, then send messages to its listener.

async function sendToTab(tabId, msg) {
  for (let retry = 0; retry < 2; retry++) {
    try {
      return await chrome.tabs.sendMessage(tabId, msg, {frameId: 0});
    } catch (err) {
      if (!err.message.includes('Receiving end does not exist')) throw err;
      await chrome.scripting.executeScript({
        target: {tabId},
        files: ['content.js'],
        injectImmediately: true,
      });
    }
  }
}

// content.js:

chrome.runtime.onMessage.addListener(msg => console.log(msg));

[Bogdan Nazaruk]

Ok, this is just frustrating. I can't make the listeners to do anything.

Manifest:

  "background": {

    "service_worker": "js/background.js"

  },

  "content_scripts": [

    {

      "matches": ["<all_urls>"],

      "js": ["js/content-script.js"]

    }

  ],

"permissions": [

    "webRequest",

    "activeTab",

    "scripting",

    "nativeMessaging"

  ]

The backround.js:

let interval = setInterval(() => {sendToTab("@@@ Debugging: Testing the messaging")},1000);

sendToTab("@@@ Debugging TEST sendToTab!");

async function sendToTab(msg) {

  const [tab] = await chrome.tabs.query({active: true, lastFocusedWindow: true});

  const tabId = tab.id;

  for (let retry = 0; retry < 20; retry++) {

    try {

      return await chrome.tabs.sendMessage(tabId, msg, {frameId: 0});

    } catch (err) {

      if (!err.message.includes('Receiving end does not exist')) throw err;

    }

  }

}

And the content-script.js:

console.log("@@@ Debugging content-script.js");

chrome.runtime.onMessage.addListener(msg => console.log("@@@ Debugging CS listener 2:", msg));

With this background script, I get the messages passed to the content-script successfully. However, once I replace the interval with an actual listener, it does completely nothing. Like, nothing at all. As if the event callback is never ever triggered. Here's the listener:

console.log("@@@ debugging backround initialized");

const filter = {urls: ["<all_urls>"]}

const urls = new Map();

chrome.webRequest.onBeforeRequest.addListener(info => {

  console.log("@@@ debugging service worker: the url is ", info.url);

  sendToTab({info: info, eventTriggered:"onBeforeRequest"});

}, filter);

Seems like I get logs from the first line, but no logs from within the callback. Callback is never triggered. Maybe I'm missing a permission to listen to all Chromes network requests? But there are completely no errors. Maybe I'm supposed to use a different method to deploy the listener? 

[Bogdan Nazaruk]

errr... I switched to work on implementing settings functionality for the extension and suddenly... My callbacks work! I have no idea why. Didn't touch the manifest. Only the extension code. I have reloaded the extension many times and it didn't work.

Well, I don't know why it works, but it does now.

[Bogdan Nazaruk]

Goooooosh.... I see what the problem was. It was not enough to just reload the extension. I had to open the popup once. 

So... I don't understand why this happens. Event listeners refuse to work until I open the popup on a given page. Why can that be? I need to fix it. It should work on every active tab.

[wOxxOm]

1. The background script (service worker) add listeners for external events such as `chrome.webRequest`, otherwise it won't wake up.
2. Your sendToTab doesn't use tabId in the event's info.
3. You also need host_permissions in manifest.json for webRequest to see the requests.

[wOxxOm]

The fact that your incorrect background.js started working when you open the popup may mean that you loaded this file in your popup's html, which is a mistake. The background script should be only declared in manifest.json's "background" section.

[Bogdan Nazaruk]

1. I guess it adds them, but I'm not sure why it waits for the popup to start working. I see the first console log firing in its service worker console so I know it runs every time I reload the extension, and its listeners code is executed then too, but no errors and silent console until I open the popup.

2. Yes, there's no need to supply sendToTab with the tabId. It does it on its own, I always only wanna send events to the active tab. This is my MVP. I don't yet see any value in sending events to other tabs, so I moved the tab from the arguments of the function inside it.

3. That's a great point. Let's elaborate a bit on it. From the host permissions documentation I didn't really get how to properly use them for my usecase, but I've had them like so:

  "host_permissions": [

    "<all_urls>",

    "https://*/*",

    "http://*/*"

  ],

  "optional_host_permissions":[

    "https://*/*",

    "http://*/*"

  ],

I'm trying to remove any limitations. Seems like Chrome doesn't throw errors on these, so they're ok. And yet! besides the liteners demanding popup to start working, I have another problem: when they do work, I only can see current url in info.url. It doesn't allow me to see third party requests. I cleared the console and sent a request from it. I can see it in the network tab, but the listeners don't react to it. Seems to me like the host permission issue. But maybe not just it.

4. I don't load background script from my popup code. It's loaded via manifest. But yeah, there must be something in the popup code that affects it. Very odd. Those event listeners. They work in the popup's console too, correctly identifying extension files the popup loads. Like it's own JS, css and stuff like that. Which is interesting, cuz  I didn't expect it to work in the popup window. But once it does, it starts working in the active tab too. Here is my popup code: https://gist.github.com/cthae/96b35f149191858d6da500bc7249b70c Nothing there seems suspicious to me. Maybe you can find a reason there?

5. And now storage! I allowed it in the manifest and trying to use it in the popup code. Seems like chrome.storage.sync.get suddenly claims sync is undefined. How come it's undefined? I literally take it from the documentation. I really didn't expect any roadblocks in implementing settings. The only reason I can come up with is that the storage api isn't allowed in the popup code? So like we're supposed to use it only in the background script? No, this can't be true. It's too absurd to be true. Settings should be usable on all three levels: popup, background and cs. Do we maybe have a table that would clarify which apis are allowed in which scopes?

V3 makes it A LOT harder. MUCH MUCH harder to do anything. I'm at a loss. I've dumped enough time into it now to start contemplating switching to v2. It feels like v3 is so raw they will need another five years to bring it to be adequately usable. Too 

many unexpected artificial limitations with seemingly little reason.

[wOxxOm]

1. Your code doesn't add any listeners, so the service worker runs only one time after [re]installation (reloading is re-installation) to gather the listeners, finds none, and never re-awakens again. You can't use it this way. You need to listen to external events i.e. explicitly call chrome.webRequest.onXXX.addListener.
   
   
2. It'll be wrong when the request is made in another non-active tab for the site.
   
   
3. "host_permissions": ["<all_urls>"] is enough as it includes the other patterns you've specified. There's no need for optional_host_permissions if you use host_permissions. Eventually you may want to migrate from host_permissions to optional_host_permissions if you decide to ask the user to allow the URL.
   
   
4. The only other explanation is that the browser loads the service worker any time it loads a page that belogs to SW-controlled origin like the popup. You see the output from the service worker in the same console because devtools considers SW to belong to all its controlled pages - this behavior makes sense for web pages as they use an onfetch event listener in their SW, but 99.9% of extensions don't use it and don't need it. In the future Chrome is likely to stop loading SW in this scenario (no onfetch listener).
   
   
5. Assuming it's in chrome://extensions Errors list, it may be an old error from the time you've tried to use chrome.storage before adding "storage" to "permissions". Another reason is loading the popup's html as a local file:// or from localhost so it won't have chrome-extension:// URL.

[Bogdan Nazaruk]

1. My background script adds listeners exactly as you've described, how else would I be able to see the network requests in the console otherwise? 

chrome.webRequest.onBeforeRequest.addListener((info) => {

  //if(/b\/ss/.test(info.url))

  sendToTab({info: info, eventTriggered:"onBeforeRequest"});

}, filter);

2. Well, I don't think this ever happens in my case. But sure, once I'm done with more pressing issues, I will see if I can get a tab id from the info object. Or how to otherwise trace the origin of the request.

3. Ah, ok. Will clean that up.

Ok, I'll just unload the whole project as is to github to make it easier to debug it cuz guessing things around is not very efficient. Here it is: https://github.com/cthae/Adobe-Launch-Debugger/tree/main

Main mysteries there are:

1. I couldn't find out why settings don't work. The errors in the extensions are fresh.

2. I can't grasp why the event listeners don't work properly until the popup opens.

3. And I can't understand why when the listeners work, they only show requests to local domain, no third party requests trigger the listener.

If you could help debugging those, it would be awesome.

[wOxxOm]

Here's the fix:

• host_permissions in your manifest.json are incorrectly nested inside externally_connectable. Pull it out.
• Add "storage" to "permissions".
  

> why the event listeners don't work properly until the popup opens

It's because you use "activeTab" permission, which temporarily grants the host permissions to the active tab once the user explicitly invokes your extension's UI.

> only show requests to local domain

Because the active tab was a local domain, to which "activeTab" granted the permissions.

P.S. My note #1 was referring to your setInterval code, which was what caused the wake up problem. With your new code that uses webRequest, assuming you reloaded the extension in chrome://extensions page, the service worker should automatically wake up on a matching event.

[Bogdan Nazaruk]

odd. So active tab in my case adds restrictions rather than permissions? I removed it and suddenly all starts looking better.

[Bogdan Nazaruk]

It's awesome. All works nicely now. I finally can reliably work with web requests! Whew! What a relief. 

Thanks a lot wOxxOm, you've helped a bunch. I feel like 80% of work is done now. And all important architectural issues are solved. 

[wOxxOm]

>  active tab in my case adds restrictions rather than permissions

No. You had placed host_permissions in the wrong place so your extension had no host permissions by default and "activeTab" gave you one temporarily.

[Bogdan Nazaruk]

Ah that makes perfect sense! Thanks a lot :)