declaratkveNetRequest modifyHeaders doesnt work at all

[avian]

[

  {

    "id": 1,

    "priority": 1,

    "condition": {

      "urlFilter": "|https*",

      "resourceTypes": ["xmlhttprequest"]

    },

    "action": {

      "type": "modifyHeaders",

      "responseHeaders": [

        { "header": "h1", "operation": "remove" },

        { "header": "h2", "operation": "set", "value": "v2" },

        { "header": "h3", "operation": "append", "value": "v3" },

        {

          "header": "referer",

          "operation": "append",

          "value": "https://mysite.com"

        }

      ]

    }

  }

]

On any request, foreground page or background page - when I check these headers are added / changed in the Network Tab - literally nothing gets added whatsoever

  "declarative_net_request": {

    "rule_resources": [

      {

        "id": "ruleset_1",

        "enabled": true,

        "path": "rules_1.json"

      }

    ]

  },

  "permissions": [

    "tabs",

    "cookies",

    "downloads",

    "storage",

    "webRequest",

    "declarativeNetRequest",

    "declarativeNetRequestFeedback",

    "contextMenus"

  ],

By the way guys, I checked - it's the NETWORK RULE is getting fired JUST FINE - I checked with 

chrome.declarativeNetRequest.onRuleMatchedDebug.addListener(function (o) {

  console.log('rule matched:', o);

});

But the FINAL RESULT, NO HEADERS ARE MODIFIED OR ADDED??

What's the point in this rule if it does LITERALLY NOTHING??

Why is manifest V3 SO crap??

[avian]

By the way...action: "block" works fine. So what gives? Can we not modify requests anymore?

Both requests: FOREGROUND + BACKGROUND don't get modified at all. NOT THE EXAMPLE ACTIONS, or MY actions that I want to take to modify the requests.

WHY?

[Cuyler Stuwe]

IIRC, it only works with an extremely limited number of very specific headers, and which headers these were wasn't easy/obvious to find.

--
You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/a7555e2d-9715-4e7c-918f-40d556ac31ddn%40chromium.org.

[Cuyler Stuwe]

E.g., there are tons of MV2 extensions in the past where I've needed to modify the X-Frame-Options response header as a fundamental requirement in order to get an extension to serve its purpose, and I don't think that's something I could change with MV3.

[Simeon Vincent]

Earlier on we planned to limit modifications to headers like X-Frame-Options, but last time I spoke with the engineers working on declarativeNetRequest they indicated that we hadn't implemented these restrictions and did not plan to. Cuyler, have you tested this recently?

Unfortunately issue 1247400 makes figuring out if your rules are behaving as expected even harder as not all modifications are reflected in DevTools.

Simeon - @dotproto
Chrome Extensions DevRel

On Tuesday, May 3, 2022 at 1:26:05 PM UTC-7 salem...@gmail.com wrote:

E.g., there are tons of MV2 extensions in the past where I've needed to modify the X-Frame-Options response header as a fundamental requirement in order to get an extension to serve its purpose, and I don't think that's something I could change with MV3.

On Tue, May 3, 2022 at 1:23 PM Cuyler Stuwe <salem...@gmail.com> wrote:

IIRC, it only works with an extremely limited number of very specific headers, and which headers these were wasn't easy/obvious to find.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.

[Cuyler Stuwe]

Most often, I’ve wanted to strip off X-Frame-Options in extensions where it makes sense to keep a website open in an invisible persistent webpage (in MV2, this would’ve been background.js).

For example, I wrote a comparison shopping site for a client whose purpose was to compare the same exact product across multiple stores “as the user sees it” (logged in to account, with all personalized promos/deals/coupons, etc). Of the handful of stores checked, it would add the item to the cart at whichever store was cheapest.

These sites didn’t have APIs that could be checked. So we fell back to a scraper strategy: In order to do this, I iframed all of the relevant stores in background.js, had each site run a content script, and had the background page emulate user actions in each frame by sending messages that each was set up to respond to.

Stripping X-Frame-Options was critical for this method to work at all.

Situations like these where you want to create a “virtual local API” via scraping and emulating user actions aren’t really supported very well by MV3, so I’ve basically tossed this approach in the trash. Haven’t had a reason to try to strip X-Frame-Options so far since.

On Tue, May 3, 2022 at 8:19 PM Simeon Vincent <sim...@chromium.org> wrote:

Earlier on we planned to limit modifications to headers like X-Frame-Options, but last time I spoke with the engineers working on declarativeNetRequest they indicated that we hadn't implemented these restrictions and did not plan to. Cuyler, have you tested this recently?

Unfortunately issue 1247400 makes figuring out if your rules are behaving as expected even harder as not all modifications are reflected in DevTools.

Simeon - @dotproto
Chrome Extensions DevRel

On Tuesday, May 3, 2022 at 1:26:05 PM UTC-7 salem...@gmail.com wrote:

E.g., there are tons of MV2 extensions in the past where I've needed to modify the X-Frame-Options response header as a fundamental requirement in order to get an extension to serve its purpose, and I don't think that's something I could change with MV3.

On Tue, May 3, 2022 at 1:23 PM Cuyler Stuwe <salem...@gmail.com> wrote:

IIRC, it only works with an extremely limited number of very specific headers, and which headers these were wasn't easy/obvious to find.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

[Simeon Vincent]

I believe X-Frame-Options striping has also been used for things like embedding video streaming services in watch party extensions. IMO extension frames should be able to ignore these kinds of restrictions so you don't have to disable the header entirely, but I'm getting off track.

FWIW, scraping pages in the background is one of the use cases I want to have on our formal list of "reasons" in offscreen documents. "DOM_SCRAPING" was even used in the short list of examples reasons in that proposal. 

Simeon - @dotproto
Chrome Extensions DevRel

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extensions+unsub...@chromium.org.

[Cuyler Stuwe]

Yeah, but a short-lived offscreen document (as has been discussed) wouldn’t really be useful for a lot of DOM scraping purposes.

A lot of .NET apps, for example, are extremely stateful and need to be held open indefinitely so as not to lose place in paginated results along with the stateful outcome of all actions taken up to a given point.

The type of scraping solutions I’ve been contracted to build by real estate agents in the past tend to fall into this bucket.

And the extensions that the Mechanical Turk community have built to poll for and grab fresh new microtasks every ~1sec definitely need something long-running.

Still waiting for a deeper acknowledgment from the team at large that there is a wide swath of legitimate applications for long-running extensions that poll, respond instantly to network events, and so on.

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

[wOxxOm]

Since devtools doesn't show the changed headers yet, you should use an external tool like Fiddler, Charles, mitmproxy, wireshark, etc.

[Simeon Vincent]

Yeah, but a short-lived offscreen document (as has been discussed) wouldn’t really be useful for a lot of DOM scraping purposes.

Offscreen documents aren't persistent, but that doesn't necessarily mean they are short lived. The current plan is that offscreen documents will live as long as necessary to fulfil their purpose*. For example, an episode of a podcast can be over an hour long, so a podcast client extension would need to have a long lived (but not necessarily persistent) page context to play that audio file. In this scenario it's conceivable that an offscreen document will stay alive for multiple hours. Scripting a page loaded in the background (I hesitate to call that scraping, but I digress) is another case where an offscreen document may end up living for an extended period.

The asterisk (*) above centers on lifetime guarantees. We intend for offscreen document lifetimes to be very similar to event page lifetimes, but with the key difference that offscreen documents may be terminated unexpectedly. With an event page, the expectation is that the page will run until it yields or closes itself. With offscreen documents, the page will run until it yields, closes itself, or is terminated without notice. That's a critical distinction because it means extensions authors will need to anticipate and plan to recover from such terminations.

Simeon - @dotproto

Chrome Extensions DevRel

--

You received this message because you are subscribed to the Google Groups "Chromium Extensions" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-extens...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/cb88b28e-1416-4aa2-8ff8-f7215bbf9f05n%40chromium.org.

[Cuyler Stuwe]

Let’s say I’m scraping a stateful SPA with 300 pages that don’t allow you to access pages by index until they’ve been primed by walking though the pages with the “next” button. The user might need a result only once per 10-30 minutes, and the result might be on a random page. The user needs the most recent result, which is polled by the server on a new page view. Would the current plan support this?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-extensions/CAFY0HLMbT-_3riC%2BMgoqtjUyBk9%3DMC27mm06YhbUs4c8xe3sXQ%40mail.gmail.com.

[hrg...@gmail.com]

 

With offscreen documents, the page will run until it yields, closes itself, or is terminated without notice.

Is this because Android can force a process to terminate at any given moment?

What's going to happen on other OSs?

Are you guys going to implement some random forceful termination in order to make that Android limitation uniform across all OSs?

[Simeon Vincent]

Let’s say I’m scraping a stateful SPA with 300 pages that don’t allow you to access pages by index until they’ve been primed by walking though the pages with the “next” button. The user might need a result only once per 10-30 minutes, and the result might be on a random page. The user needs the most recent result, which is polled by the server on a new page view. Would the current plan support this?

Yeah, I think it would. That said, I'd be curious to dig a bit more into the expected user experience and intended features for this extension. Based on what you've shared so far it's not clear that this situation requires background scraping, so I'm assuming there are other factors at play.

Simeon - @dotproto

Chrome Extensions DevRel

[Cuyler Stuwe]

“Require” is a sticky word here. Most scraping extensions don’t “require” things to run in the background, until “offering a good/intuitive user experience” becomes one of the requirements. Nobody on Earth wants to clutter their tabs with processes doing automated programmatic things they shouldn’t have to care about and have to remember not to close.

[Cuyler Stuwe]

Sure, I can meet.