Intent to Ship: Changes to the XSS Auditor.

[Mike West]

# Contact Emails

tse...@chromium.org, mk...@chromium.org

# Spec

None. `X-XSS-Protection` is a pile of heuristics that differs from browser to browser, and has never been documented outside of the source code.

https://msdn.microsoft.com/en-us/library/dd565647(v=vs.85).aspx is about as close as you'll get.

# Summary

By default, Chrome acts as though an `X-XSS-Protection: 1` header is present, and intervenes against scripts that appear to be reflected from the address the user has navigated to. Detected reflections are filtered out by ignoring the entire script block or event handler in which they appear.

We'd like to make two changes:

1. Change the default behavior to `X-XSS-Protection: 1; mode=block`, which blocks the page load by navigating to a unique origin when XSS is detected, rather than filtering out specific scripts.

2. Deprecate the filter mode, with the intent to remove it completely at some future date.

# Motivation

Filtering is in some ways the worst of all possible worlds. It gives an attacker the ability to disable specific scripts on a page in a way that an unsuspecting page can't mitigate. Savvy developers already opt-into blocking mode, or disable the auditor altogether. See https://blog.innerht.ml/the-misunderstood-x-xss-protection/ for some examples.

Our default stance should be secure; blocking the page load ensures that we don't create new issues for developers while intervening to "fix" reflected XSS.

(Some folks on Google's security team would prefer that we remove the auditor entirely. I don't think that's a good idea for a number of reasons, but it's an option to consider.)

# Interop Risk

Low, but only because there's no spec, and every browser that supports an XSS auditor (IE, Edge, Safari, Opera/Chrome) do things differently, according to their own pile of heuristics. Since none of us have taken the time to write down the auditor's mechanisms, there's already practically zero interop.

This change would move from zero interop to zero interop, which isn't great, but also isn't a huge step backwards.

# Compat Risk

Pages that are currently partially broken due to the XSS Auditor will be totally broken due to the auditor. That is, if we're currently blocking some script on your pages due to false positives, and you haven't already fixed it by opting out of the auditor (via `X-XSS-Protection: 0`), then we'll block the page entirely.

This is a risk, certainly. However, the fact that affected pages are already broken in one way or another mitigates the risk. Raising the visibility of the breakage will help developers either fix their pages, or restore complete functionality by disabling the auditor.

# Technical Constraints

None.

# All Blink Platforms?

Yes.

# OWP Bug

https://crbug.com/654794

# Chromestatus

https://www.chromestatus.com/features/5748927282282496

# Requesting approval to ship?

Requesting approval to ship #1: changing the default.

Requesting approval to deprecate for #2: deprecating filtering.

-mike

[Jochen Eisinger]

blocking rather than crippling the page by default sounds like a good trade-off, so lgtm1

[Kouhei Ueno]

non-owner LGTM from a html/parser maintainer.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

--

kouhei

[Rick Byers]

Do you have any stats on how often we'd be blocking loads (either UMA or via some crawl)?  It would be nice to put an upper bound on the possible breakage.  I agree that it's generally going to be better to block the page load than filter the script.  But if, for example, there was some popular analytics library that was getting blocked on many sites without any user-visible breakage, then we might need to do some outreach in advance of breaking all those sites.  If there's data indicating this is very rare, then I'm happy proceeding without any further compat analysis.

Rick

[Ojan Vafai]

Also, do you have data on how many page views have "X-XSS-Protection: 0"? That can be a good measure of how much the current thing is worked around.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--

kouhei

[isaac....@gmail.com]

I ran a scan of the top 1 million websites last month here is what I saw for X-XSS-Protection:
total,56784
0,1828
0; report,0
0; mode=block,0
invalid setting,55
1,2003
1; report,54
1; mode=block,47370
1; mode=block report,5427

[isaac....@gmail.com]

[Mike West]

On Wed, Oct 12, 2016 at 5:41 AM, Ojan Vafai <oj...@chromium.org> wrote:

Also, do you have data on how many page views have "X-XSS-Protection: 0"? That can be a good measure of how much the current thing is worked around.

We've only just added basic metrics to the auditor, unfortunately.

In Canary, ~4% of page views over the last week opted into blocking mode (Google opts into blocking, for example), ~1.5% of page views opted out of the protection (Facebook opts out, for example). The remaining ~94.5% accept the default.

On Tue, Oct 11, 2016 at 7:58 PM Rick Byers <rby...@chromium.org> wrote:

Do you have any stats on how often we'd be blocking loads (either UMA or via some crawl)? 

Over the last week in Canary, we blocked 15 pages, and filtered script out of 704. Canary, of course, has a tiny population, and is completely unrepresentative of anything. I can report back to y'all later once those metrics hit a larger population.

 

It would be nice to put an upper bound on the possible breakage.  I agree that it's generally going to be better to block the page load than filter the script.  But if, for example, there was some popular analytics library that was getting blocked on many sites without any user-visible breakage, then we might need to do some outreach in advance of breaking all those sites.  If there's data indicating this is very rare, then I'm happy proceeding without any further compat analysis.

I think (hope!) this is unlikely. The complaints we get about the auditor are that it's not magical enough in terms of blocking various reflection techniques. I don't know when the last false-positive bug we got was... Tom would.

-mike

[Philip Jägenstedt]

In other words, put this on hold for a bit until more data is available?

[Mike West]

On Wed, Oct 12, 2016 at 4:30 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

In other words, put this on hold for a bit until more data is available?

If it makes you more comfortable to wait for the metrics to hit stable, sure.

That said, the XSS metrics are in the latest dev build, so let me give you a little more data from the last 7 days on that channel:

* We filtered reflected script 2,617 times.

* We blocked the entire page 16 times.

* ~2.1% of page views turned the auditor off.

* ~0.04% of page views turned the auditor on in filter mode (~no-op).

* ~4.8% of page views turned the auditor on in block mode.

* ~0.009% of page views had an invalid header value.

Rick, one note: the data from the hot new metric (https://crbug.com/597963) is radically different. According to that metric:

* We filtered reflected script 3,254 times.

* We blocked the entire page 19 times.

* ~6.5% of page views turned the auditor off.

* ~0.1% of page views turned the auditor on in filter mode (~no-op).

* ~17.5% of page views turned the auditor on in block mode.

* ~0.02% of page views had an invalid header value.

These numbers show the same general trends (we're not blocking many pages, and opting-into block mode is 2-3x as popular as opting-out of the auditor entirely), but the order of magnitude difference in some metrics surprises me. Might be worth digging in to see what's going on.

-mike

[Artur Janc]

Just as a note, Google enables the behavior Mike is talking about (mode=block) at the framework level and we've been setting it for almost all our applications for many years. We've also been hit by IE-specific XSS-es in a few places where we forgot to enable blocking mode and the default filtering behavior kicked in.

It seems like a good idea to go forward with this.

[Rick Byers]

On Thu, Oct 13, 2016 at 10:39 AM, Artur Janc <a...@google.com> wrote:

Just as a note, Google enables the behavior Mike is talking about (mode=block) at the framework level and we've been setting it for almost all our applications for many years. We've also been hit by IE-specific XSS-es in a few places where we forgot to enable blocking mode and the default filtering behavior kicked in.

It seems like a good idea to go forward with this.

On Thu, Oct 13, 2016 at 9:18 AM, Mike West <mk...@chromium.org> wrote:

On Wed, Oct 12, 2016 at 4:30 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

In other words, put this on hold for a bit until more data is available?

If it makes you more comfortable to wait for the metrics to hit stable, sure.

If the metrics from dev channel suggest the compat risk is low, I'm fine making tentative decisions based on that (and just double-checking once we have beta or stable metrics that it's not radically different).

That said, the XSS metrics are in the latest dev build, so let me give you a little more data from the last 7 days on that channel:

* We filtered reflected script 2,617 times.

So for the compat risk of the proposed change, it's primarily this value that matters, right?  Virtually all of these will be turned into blocking the entire page.  Is this a FeatureObserver bucket or some other histogram?  Just trying to put the number in context (eg. as % of PageVisits).

* We blocked the entire page 16 times.

* ~2.1% of page views turned the auditor off.

* ~0.04% of page views turned the auditor on in filter mode (~no-op).

* ~4.8% of page views turned the auditor on in block mode.

* ~0.009% of page views had an invalid header value.

Rick, one note: the data from the hot new metric (https://crbug.com/597963) is radically different. According to that metric:

Yeah, we know that the old FeatureObserver.PageVisits was under-counting such that the absolute values are likely 2x-3x higher than reported.  But really it's the relative value that matters since we're calibrated against it (in terms of what's worked on well in practice).  We probably shouldn't worry too much about the new numbers for intents yet until I've finished improving them and convincing ourselves of their accuracy.  Then we can re-calibrate our expectations and switch over (predictability has a Q4 OKR for this).

[Mike West]

On Thu, Oct 13, 2016 at 5:08 PM, Rick Byers <rby...@chromium.org> wrote:

On Thu, Oct 13, 2016 at 10:39 AM, Artur Janc <a...@google.com> wrote:

Just as a note, Google enables the behavior Mike is talking about (mode=block) at the framework level and we've been setting it for almost all our applications for many years. We've also been hit by IE-specific XSS-es in a few places where we forgot to enable blocking mode and the default filtering behavior kicked in.

It seems like a good idea to go forward with this.

On Thu, Oct 13, 2016 at 9:18 AM, Mike West <mk...@chromium.org> wrote:

On Wed, Oct 12, 2016 at 4:30 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

In other words, put this on hold for a bit until more data is available?

If it makes you more comfortable to wait for the metrics to hit stable, sure.

If the metrics from dev channel suggest the compat risk is low, I'm fine making tentative decisions based on that (and just double-checking once we have beta or stable metrics that it's not radically different).

SGTM.

 

That said, the XSS metrics are in the latest dev build, so let me give you a little more data from the last 7 days on that channel:

* We filtered reflected script 2,617 times.

So for the compat risk of the proposed change, it's primarily this value that matters, right?  Virtually all of these will be turned into blocking the entire page.  Is this a FeatureObserver bucket or some other histogram?  Just trying to put the number in context (eg. as % of PageVisits).

This is `UseCounter::XSSAuditorBlockedScript`. It turns into something like 0.0003% of page views in the old metric, and 0.0007% in the new metric that I'm not worrying about until later. :)

[Philip Jägenstedt]

What kinds of pages are typically blocked? Out of that small number, can you make any guess as to how many will translate to sad users?

[Rick Byers]

On Thu, Oct 13, 2016 at 11:40 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

What kinds of pages are typically blocked? Out of that small number, can you make any guess as to how many will translate to sad users?

On Thu, Oct 13, 2016 at 5:16 PM 'Mike West' via blink-dev <blin...@chromium.org> wrote:

On Thu, Oct 13, 2016 at 5:08 PM, Rick Byers <rby...@chromium.org> wrote:

On Thu, Oct 13, 2016 at 10:39 AM, Artur Janc <a...@google.com> wrote:

Just as a note, Google enables the behavior Mike is talking about (mode=block) at the framework level and we've been setting it for almost all our applications for many years. We've also been hit by IE-specific XSS-es in a few places where we forgot to enable blocking mode and the default filtering behavior kicked in.

It seems like a good idea to go forward with this.

On Thu, Oct 13, 2016 at 9:18 AM, Mike West <mk...@chromium.org> wrote:

On Wed, Oct 12, 2016 at 4:30 PM, Philip Jägenstedt <foo...@chromium.org> wrote:

In other words, put this on hold for a bit until more data is available?

If it makes you more comfortable to wait for the metrics to hit stable, sure.

If the metrics from dev channel suggest the compat risk is low, I'm fine making tentative decisions based on that (and just double-checking once we have beta or stable metrics that it's not radically different).

SGTM.

 

That said, the XSS metrics are in the latest dev build, so let me give you a little more data from the last 7 days on that channel:

* We filtered reflected script 2,617 times.

So for the compat risk of the proposed change, it's primarily this value that matters, right?  Virtually all of these will be turned into blocking the entire page.  Is this a FeatureObserver bucket or some other histogram?  Just trying to put the number in context (eg. as % of PageVisits).

This is `UseCounter::XSSAuditorBlockedScript`. It turns into something like 0.0003% of page views in the old metric, and 0.0007% in the new metric that I'm not worrying about until later. :)

Ah, perfect - thanks.  This is exactly what I was looking for - a reasonably low bound on the compat risk.

When a page is blocked I assume developers will see an error (at least when reproducing with the inspector open) which they can search on to get advice like this showing how to disable the XSS auditor, right?  Any doc / guidance updates you think we need to take this change into account?  We may want to mention this in Joe's "deprecations and removes for Chrome 56" articles.

All of these cases would have previously had console warnings about the blocked scripts, so I think they've effectively already gone through a "deprecation period" where we've given notice of the problem.

Given these three properties, I'm now convinced that we've done the necessary due diligence to mitigate the compat risk.  LGTM1 to ship.

[Mike West]

On Thu, Oct 13, 2016 at 7:51 PM, Rick Byers <rby...@chromium.org> wrote:

On Thu, Oct 13, 2016 at 11:40 AM, Philip Jägenstedt <foo...@chromium.org> wrote:

What kinds of pages are typically blocked? Out of that small number, can you make any guess as to how many will translate to sad users?

Ideally, we filter script where an attacker has tricked a user into clicking a link that contains data that's reflected into the document's HTML. In false-positive cases, we would filter a script on a page that contained data in the URL that happened to look like some of the page's content. Tom's pretty good about not doing that, though. We haven't had reports of false positives that I remember.

This is `UseCounter::XSSAuditorBlockedScript`. It turns into something like 0.0003% of page views in the old metric, and 0.0007% in the new metric that I'm not worrying about until later. :)

Ah, perfect - thanks.  This is exactly what I was looking for - a reasonably low bound on the compat risk.

When a page is blocked I assume developers will see an error (at least when reproducing with the inspector open) which they can search on to get advice like this showing how to disable the XSS auditor, right?  Any doc / guidance updates you think we need to take this change into account?  We may want to mention this in Joe's "deprecations and removes for Chrome 56" articles.

We should mention it in an article, and folks do get a console error. I don't think that's enough, though: we should have an error page, and I have most of a CL together to do that. I don't plan on flipping the default until we land folks on something other than a blank white page.

 

Given these three properties, I'm now convinced that we've done the necessary due diligence to mitigate the compat risk.  LGTM1 to ship.

LGTM2, I think, since Jochen already LGTM1'd? Thanks!

[Jochen Eisinger]

right, still lgtm

[Rick Byers]

Perfect.  We should probably plan on having some way for developers to connect this error page to the article that has more details.  Is a "more info" link the right way to go (even though most people who see it will be end users with no understanding/interest in the developer details)?  Or perhaps you just want to make sure the article includes the key words from the error message so that a google search for the error message is likely to land you on the article?

 

Given these three properties, I'm now convinced that we've done the necessary due diligence to mitigate the compat risk.  LGTM1 to ship.

LGTM2, I think, since Jochen already LGTM1'd? Thanks!

Whoops, sorry! 

[Philip Jägenstedt]

Thanks Mike, LGTM3.

[Mike West]

This didn't make the 56 branch point; the error page took a bit longer to hammer out than I expected, but it looks like we've converged on something reasonable in https://codereview.chromium.org/2425663002/. Once that lands, I intend to change the behavior for 57.

I've updated the status entry accordingly: https://www.chromestatus.com/feature/5748927282282496

-mike