Intent to Deprecate and Remove: Private Network Access requests for subresources without proper preflight response

[Jonathan Hao]

Context

Since M104, we've started sending preflight requests before private network access, but ignoring the preflight result (or the lack of it). After analyzing the URL-keyed metrics, we found that most of them don't seem legit, most likely used for fingerprinting purposes, so we decided to start enforcing the preflight response, which means that the websites will not be able to fetch subresources from less-public ip address space without getting proper preflight responses.

An intent focusing on the deprecation trial was sent earlier in this thread: https://groups.google.com/a/chromium.org/g/blink-dev/c/k8osI88QbKs/m/16ytAQ-BAwAJ?utm_medium=email&utm_source=footer.  There, we were suggested to send a separate intent to remove the functionality, and hence this intent email.

Contact emails

tit...@chromium.org, va...@chromium.org, cl...@chromium.org, l...@chromium.org, ph...@chromium.org

Explainer

https://github.com/WICG/private-network-access/blob/main/explainer.md

Specification

https://wicg.github.io/private-network-access

Design docs

https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit

Summary

Sends a CORS preflight request ahead of any private network requests for subresources, asking for explicit permission from the target server. A private network request is any request from a public website to a private IP address or localhost, or from a private website (e.g. intranet) to localhost. Sending a preflight request mitigates the risk of cross-site request forgery attacks against private network devices such as routers, which are often not prepared to defend against this threat.

Blink component

Blink>SecurityFeature>CORS>PrivateNetworkAccess

TAG review

https://github.com/w3ctag/design-reviews/issues/572

TAG review status

Issues addressed

Risks

Interoperability and Compatibility

The main interoperability risk, as always, is if other browser engines do not implement this. Compat risk is straightforward: web servers that do not handle the new preflight requests will eventually break, once the feature ships. The plan to address this is as follows: 1. Send preflight request, ignore result, always send actual request. Failed preflight requests will result in a warning being shown in devtools. 2. Wait for 3 milestones. 3. Gate actual request on preflight request success, with deprecation trial for developers to buy some more time. 4. End deprecation trial 4 milestones later. UseCounters: https://chromestatus.com/metrics/feature/timeline/popularity/3753 https://chromestatus.com/metrics/feature/timeline/popularity/3755 https://chromestatus.com/metrics/feature/timeline/popularity/3757 The above measure pages that make at least one private network request for which we would now send a preflight request.

Gecko: Worth prototyping (https://github.com/mozilla/standards-positions/issues/143)

WebKit: No signal (https://lists.webkit.org/pipermail/webkit-dev/2021-November/032040.html) Pending response.

Web developers: No signals Anecdotal evidence so far suggests that most web developers are OK with this new requirement, though some do not control the target endpoints and would be negatively impacted.

Other signals:

Ergonomics

None.

Activation

Gating access to the private network overnight on preflight requests would likely result in widespread breakage. This is why the plan is to first send requests but not act on their result, giving server developers time to implement code handling these requests. Deprecation warnings will be surfaced in DevTools to alert web/client developers when the potential for breakage later on is detected. Enforcement will be turned on later (aiming for 3 milestones), along with a deprecation trial for impacted web developers to buy themselves some more time. Experience suggests a large fraction of developers will not notice the advance deprecation warnings until things break.

Security

This change aims to be security-positive, preventing CSRF attacks against soft and juicy targets such as router admin interfaces. It does not cover navigation requests and workers, which are to be addressed in followup launches. DNS rebinding threats were of particular concern during the design of this feature: https://docs.google.com/document/d/1FYPIeP90MQ_pQ6UAo0mCB3g2Z_AynfPWHbDnHIST6VI/edit#heading=h.189j5gnadts9

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Not going to ship on Android WebView

Goals for experimentation

Give websites time to make sure they respond to the preflights

Reason this experiment is being extended

N/A

Ongoing technical constraints

N/A

Debuggability

Relevant information (client and resource IP address space) is already piped into the DevTools network panel. Deprecation warnings and errors will be surfaced in the DevTools issues panel explaining the problem when it arises.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Not on Android WebView given previous difficulty in supporting PNA changes due to the lack of support for deprecation trials. Support for WebView will be considered separately.

Is this feature fully tested by web-platform-tests?

Yes

DevTrial instructions

https://github.com/WICG/private-network-access/blob/main/HOWTO.md

Flag name

PrivateNetworkAccessRespectPreflightResults

Requires code in //chrome?

False

Tracking bug

https://crbug.com/591068

Launch bug

https://crbug.com/1274149

Estimated milestones

OriginTrial desktop last	112
OriginTrial desktop first	109
DevTrial on desktop	98

OriginTrial Android last	112
OriginTrial Android first	109
DevTrial on Android	98

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5737414355058688

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/PrB0xnNxaHs/m/jeoxvNjXCAAJ
Intent to Ship: https://groups.google.com/a/chromium.org/g/blink-dev/c/72CK2mxD47c

This intent message was generated by Chrome Platform Status.

[Daniel Bratell]

I'm curious about the enterprise situation here. This seems to me like something that could be in use in enterprise applications, and we would not really know about it. ( https://www.chromium.org/developers/enterprise-changes/ is a good checklist for this)

Is there an enterprise policy for this that can be used for those that need the old behaviour, if not, could one be added?

Also, have you reached out to the enterprise community? (See link above)

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOC%3DiP%2BEcXFNTOfg829uzFh2YMov%3DTsmAzdP9VDn8MoLuHqjog%40mail.gmail.com.

[Titouan Rigoudy]

There are two enterprise policies for this [1]! One for disabling it entirely, and one for disabling it per origin.

We have been engaging with the enterprise team since the start of PNA, a dozen or so milestones ago. This has been announced repeatedly in enterprise release notes over the past several milestones.

Cheers,

Titouan

[1] https://developer.chrome.com/blog/private-network-access-preflight/#disable-with-enterprise-policy

[Chris Harrelson]

Thanks Titouan, two of them is definitely enough, no need for three. :)

Another question: have you done an analysis of how much of the UseCounter traffic is "illegitimate use"? Two of them are at 0.1% or 0.2%, which is a risky level. But hopefully the percentage of "legitimate" traffic is a lot lower?

Chris

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9dKw30wJL3Aj5OOmYeT5Gxy1Bb7Yf%2B7VFEmAjAEyySFWw%40mail.gmail.com.

[Titouan Rigoudy]

Sure! I went through the UKM data from Android (where we can be reasonably sure that extensions are not messing with the data) and classified a bunch of the top websites. I ended up classifying ~2/3 of the traffic in volume. Out of that, it seemed that ~7% was legitimate. There was no indication that the rest of the traffic would strongly buck that trend, I was just hitting diminishing returns for low usecount websites.

Cheers,
Titouan

[Yoav Weiss]

So if the use counters are at 0.1-0.2%, we can expect only 0.007-0.014% to be legitimate (roughly speaking)?

[Titouan Rigoudy]

Exactly! Somewhere in the ballpark of 0.01% is my rough estimate.

Cheers,

Titouan

[Daniel Bratell]

We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting today and we are still not certain about the compatibility situation. The use counters are lower after compensating for likely malicious usage but they are still not quite as low as we'd like them to be.

One thing we considered was whether the data you analyzed is representative for all platforms or whether Android is different in some way. We don't know. Do you know? If not, could you take a look at the Desktop data and check that it's the same pattern as for Android?

Another question is the likely impact of a failed request. That is a question that is much harder to answer, but maybe you have an idea?

A third question is whether the situation is likely to improve or worsen over time. Do you have any insights about that?

/Daniel

[Titouan Rigoudy]

Thanks for getting back to us!

On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <brat...@gmail.com> wrote:

We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting today and we are still not certain about the compatibility situation. The use counters are lower after compensating for likely malicious usage but they are still not quite as low as we'd like them to be.

One thing we considered was whether the data you analyzed is representative for all platforms or whether Android is different in some way. We don't know. Do you know? If not, could you take a look at the Desktop data and check that it's the same pattern as for Android?

It's a good question! I'll take a look and see what I can tell from the data.

Another question is the likely impact of a failed request. That is a question that is much harder to answer, but maybe you have an idea?

In the case of malicious usage, a failed request is a win for user security and privacy! :) Seriously though, typically it means one of two things: a) the core functionality of the page does not work, because its whole purpose was to talk to a private network server, or b) some subresources on the page fail to load because of weirdness around servers with multiple IP addresses (some private, some public) or VPN usage (site used to be public, now is on a private IP). In the case of b), a reload usually fixes the issue. 

A third question is whether the situation is likely to improve or worsen over time. Do you have any insights about that?

Optimistically, the situation should improve as a result of developers noticing the warnings we've been flashing in DevTools for a while. Pessimistically (and unfortunately this has been more our experience so far), the situation will stagnate and/or get worse until the deprecation starts, and people sign up for the deprecation trial. We can always invest some more in classifying uses and reaching out to many small websites, it's simply a tradeoff of time and effort spent trying to get ahead of the issue vs patching the security hole earlier and getting on to fixing other things.

Cheers,

Titouan 

[Titouan Rigoudy]

Hi all,

Thanks for your patience, I was travelling last week, then out for a couple days, and was unable to spend much time on this until now.

On Wed, Oct 12, 2022 at 6:35 PM Titouan Rigoudy <tit...@google.com> wrote:

Thanks for getting back to us!

On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <brat...@gmail.com> wrote:

We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting today and we are still not certain about the compatibility situation. The use counters are lower after compensating for likely malicious usage but they are still not quite as low as we'd like them to be.

One thing we considered was whether the data you analyzed is representative for all platforms or whether Android is different in some way. We don't know. Do you know? If not, could you take a look at the Desktop data and check that it's the same pattern as for Android?

It's a good question! I'll take a look and see what I can tell from the data.

The data on Desktop suggests that most usage is legitimate. ChromeOS usage is dominated by educational websites: school districts and the like. This could be driven by large providers like Clever or VPN usage. On Windows and Mac we mostly see intranet-looking websites (CRMs, portals). On Linux, we see some of the former and mostly the latter.

On desktop platforms in general, the UseCounter indicates ~0.1% (Linux, ChromeOS) to ~0.2% (Windows, MacOS) of page visits are affected.

If that's too much, we could consider shipping on Android first, where our previous analysis suggests the impact is an order of magnitude lower. We could also ship on desktop to beta only for a few milestones, in order to give websites more time to notice the change.

Cheers,

Titouan

[Rick Byers]

On Wed, Nov 2, 2022 at 12:05 PM 'Titouan Rigoudy' via blink-dev <blin...@chromium.org> wrote:

Hi all,

Thanks for your patience, I was travelling last week, then out for a couple days, and was unable to spend much time on this until now.

On Wed, Oct 12, 2022 at 6:35 PM Titouan Rigoudy <tit...@google.com> wrote:

Thanks for getting back to us!

On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <brat...@gmail.com> wrote:

We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting today and we are still not certain about the compatibility situation. The use counters are lower after compensating for likely malicious usage but they are still not quite as low as we'd like them to be.

One thing we considered was whether the data you analyzed is representative for all platforms or whether Android is different in some way. We don't know. Do you know? If not, could you take a look at the Desktop data and check that it's the same pattern as for Android?

It's a good question! I'll take a look and see what I can tell from the data.

The data on Desktop suggests that most usage is legitimate. ChromeOS usage is dominated by educational websites: school districts and the like. This could be driven by large providers like Clever or VPN usage. On Windows and Mac we mostly see intranet-looking websites (CRMs, portals). On Linux, we see some of the former and mostly the latter.

On desktop platforms in general, the UseCounter indicates ~0.1% (Linux, ChromeOS) to ~0.2% (Windows, MacOS) of page visits are affected.

If that's too much, we could consider shipping on Android first, where our previous analysis suggests the impact is an order of magnitude lower. We could also ship on desktop to beta only for a few milestones, in order to give websites more time to notice the change.

Yikes, that sure sounds like a huge amount of breakage, and so not something likely to survive a launch (partner escalations, negative press, etc.).  In fact that seems suspiciously high to me. You're saying that one in 500 public Internet pages loaded on Windows has a legitimate need to load a sub-resource from a private IP space? That's a HUGE amount, hopefully it's over-counting somehow by a few orders of magnitude :-)

In terms of risk reduction, launching first on Android is one option, but that sounds like it still has a non-trivial level of risk too. Here's a few other ideas to possibly help form a pragmatic launch strategy (mostly drawn from experience captured in bit.ly/blink-compat).

• Survey some of the legitimate looking cases in the data and see if you can reproduce the breakage. How severe is it in practice?
• Pick a random set of impacted sites and work with partnerships / gtech teams to do targeted outreach. Is it easy to get sites to fix issues? Can we learn anything about how to help make it easier for them, and/or how to filter out hits from our data that aren't actually a problem in practice?
• Do a test deployment to beta channel and see what level of feedback we get. Plan for a finch trial on stable of <1% of users (but IMHO current data suggests we're not ready even for a small stable trial).
• Ensure failures are hooked up to NEL (Network Error Logging) and work with some bigger enterprises (eg. Google) to monitor, classify and debug breakage in their environment

Cheers,

Titouan

Another question is the likely impact of a failed request. That is a question that is much harder to answer, but maybe you have an idea?

In the case of malicious usage, a failed request is a win for user security and privacy! :) Seriously though, typically it means one of two things: a) the core functionality of the page does not work, because its whole purpose was to talk to a private network server, or b) some subresources on the page fail to load because of weirdness around servers with multiple IP addresses (some private, some public) or VPN usage (site used to be public, now is on a private IP). In the case of b), a reload usually fixes the issue. 

A third question is whether the situation is likely to improve or worsen over time. Do you have any insights about that?

Optimistically, the situation should improve as a result of developers noticing the warnings we've been flashing in DevTools for a while.

FWIW, past experiences for major breaking changes suggest that this is not particularly effective. "hopeful deprecations" have pretty much never worked out :-).

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPATO9eW%3DAvds4MTkgez7rVdF9gt1GrD9kmL-YqQV4YBpGSniw%40mail.gmail.com.

[Titouan Rigoudy]

Hi Rick,

Thanks for the suggestions and constructive feedback!

We are going to exempt same-origin fetches and see what difference this makes in the data: crbug.com/1382068. We expect this to help split-horizon DNS / VPN users, and will report back with updated metrics.

On Fri, Nov 4, 2022 at 4:28 PM Rick Byers <rby...@chromium.org> wrote:

On Wed, Nov 2, 2022 at 12:05 PM 'Titouan Rigoudy' via blink-dev <blin...@chromium.org> wrote:

Hi all,

Thanks for your patience, I was travelling last week, then out for a couple days, and was unable to spend much time on this until now.

On Wed, Oct 12, 2022 at 6:35 PM Titouan Rigoudy <tit...@google.com> wrote:

Thanks for getting back to us!

On Wed, Oct 12, 2022 at 6:11 PM Daniel Bratell <brat...@gmail.com> wrote:

We (Yoav, Philip, MikeT and I) discussed this at the API OWNERS meeting today and we are still not certain about the compatibility situation. The use counters are lower after compensating for likely malicious usage but they are still not quite as low as we'd like them to be.

One thing we considered was whether the data you analyzed is representative for all platforms or whether Android is different in some way. We don't know. Do you know? If not, could you take a look at the Desktop data and check that it's the same pattern as for Android?

It's a good question! I'll take a look and see what I can tell from the data.

The data on Desktop suggests that most usage is legitimate. ChromeOS usage is dominated by educational websites: school districts and the like. This could be driven by large providers like Clever or VPN usage. On Windows and Mac we mostly see intranet-looking websites (CRMs, portals). On Linux, we see some of the former and mostly the latter.

On desktop platforms in general, the UseCounter indicates ~0.1% (Linux, ChromeOS) to ~0.2% (Windows, MacOS) of page visits are affected.

If that's too much, we could consider shipping on Android first, where our previous analysis suggests the impact is an order of magnitude lower. We could also ship on desktop to beta only for a few milestones, in order to give websites more time to notice the change.

Yikes, that sure sounds like a huge amount of breakage, and so not something likely to survive a launch (partner escalations, negative press, etc.).  In fact that seems suspiciously high to me. You're saying that one in 500 public Internet pages loaded on Windows has a legitimate need to load a sub-resource from a private IP space? That's a HUGE amount, hopefully it's over-counting somehow by a few orders of magnitude :-)

In terms of risk reduction, launching first on Android is one option, but that sounds like it still has a non-trivial level of risk too. Here's a few other ideas to possibly help form a pragmatic launch strategy (mostly drawn from experience captured in bit.ly/blink-compat).

• Survey some of the legitimate looking cases in the data and see if you can reproduce the breakage. How severe is it in practice?

We've tried this and come up short. None of the websites display the affected behavior, and most are hidden behind login pages. I tend to believe that the users of affected websites on desktop are seeing issues due to VPN use. For example, we've seen users run into issues when using Zscaler in crbug.com/1352591. We are hoping to alleviate this by exempting same-origin fetches as mentioned above.

• Pick a random set of impacted sites and work with partnerships / gtech teams to do targeted outreach. Is it easy to get sites to fix issues? Can we learn anything about how to help make it easier for them, and/or how to filter out hits from our data that aren't actually a problem in practice?

On Android, we tried reaching out but usage was dominated by websites we could not engage with. We could try on desktop! 

• Do a test deployment to beta channel and see what level of feedback we get. Plan for a finch trial on stable of <1% of users (but IMHO current data suggests we're not ready even for a small stable trial).

I think rolling out to beta only is the best next course of action at this point, if exempting same-origin fetches does not result in a big reduction of the estimated compatibility impact.

• Ensure failures are hooked up to NEL (Network Error Logging) and work with some bigger enterprises (eg. Google) to monitor, classify and debug breakage in their environment

We also considered this. Unfortunately this type of reporting would leak cross-origin information to the initiator website, so we scrapped it :(

Cheers,

Titouan

[Rick Byers]

Thanks for the follow-up. It sounds like there's a bunch of steps that can be taken to both reduce the amount of real breakage and perhaps better understand and quantify the level of breakage. I hope we can find a configuration that indicates <0.001% of page views would be broken in a user-observable way before rolling this out broadly (eg. to beta or 1% of stable users). 

Good luck and let us know if there's anything API owners can do to help work through the risk analysis!

  Rick

[Rick Byers]

Just wanted to ask more about this. I forget the details for NEL (IIRC they're complex), but for the simpler case of Depreciation Reports (see deprecation.cc), I think it handles this automatically. Deprecation reports go only to the reporting endpoints specified by the frame which triggered the report. That doesn't help the top level origin learn about breakage in cross-origin iframes they embed, but that's probably OK since it's most important to know about deprecated behavior under your direct control. Would that work here?

 We've heard from some partners that deprecation reports are helpful to them in identifying likely upcoming breakage. Of course if much of the breakage is a result of VPNs proxying http traffic through private IPs, then such reports would be confusing and unactionable by site owners. So perhaps it's best to get to the bottom of those cases first anyway?