Intent to Prototype: Deprecate support for URLs with non-IPv4 hostnames ending in numbers
Matt Menke
Contact emails
mme...@google.comExplainer
NoneSpecification
https://url.spec.whatwg.org/Summary
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid, and looked up via DNS (e.g., http://foo.127.1/). Per the Public Suffix List spec, the eTLD+1 of the hostname in that URL should be "127.1". If that is ever fed back into a URLs, "http://127.1/" is mapped to "http://127.0.0.1/" by the URL spec, which seems potentially dangerous. "127.0.0.0.1" could also potentially be used to confuse users. We want to reject URLs with these hostnames.
Blink component
Internals>Network>DNSMotivation
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid hostnames, and looked up via DNS. Example hostnames: 127.0.0.0.1, foo.0.1, 10.0.0.09, 08.1.2.3. These can be problematic for the following reason: * "http://foo.127.1/" has an eTLD+1 of "127.1", per the public suffix list spec. If that's ever used as the hostname in a new URL, however, as in "http://127.1", it will then get mapped to "http://127.0.0.1/", per the URL spec, which is a different host, which is not safe. * "http://127.0.0.0.1" and "http://1.2.3.09", both of which are looked up via DNS rather than failing or being treated as IPv4 hostnames, also seem potentially confusing. While no exploit is currently known here, we want to remove support for these as a preventative security measure. The URL spec has been updated so that any URL with a hostname ending in a number that's not an IPv4 address (including, e.g., http://foo.1./, but not http://foo.1../) is considered invalid. Since this is part of the URL spec, not the DNS spec, we want to reject these URLs are the GURL layer, for URLs with appropriate protocols (http, https, ws, wss, file). For consistency, we should also fail DNS lookup attempts of these sorts of hostnames.
Initial public proposal
https://github.com/whatwg/url/pull/619TAG review
Not required for an Intent to Deprecate, I believe.TAG review status
Not applicableRisks
Interoperability and Compatibility
Any URL with an affected hostname will fail to load, and will need to be migrated to another hostname. URLs of this form do appear to be in use, though it's not clear under what circumstances. No entry in the public suffix list is affected. Affected URLs make up no more than 0.0003% of hostnames looked up via the host resolver on any platform, and are basically not used in any file URLs, according to our metrics. On OSX and Android, about 90% of host resolver lookups for these hostnames succeed, 60% do on Linux, and 2% on Windows and ChromeOS. To allow for emergency disabling in case of wider than expected breakage, I intend to add a feature for it, and do a 50% field trial on pre-release channels, though plan to just enable the feature, rather than do a gradual rollout to stable, given the low usage.
Gecko: Positive (https://github.com/whatwg/url/pull/619#issuecomment-890826499)
WebKit: In development (https://bugs.webkit.org/show_bug.cgi?id=228826)
Web developers: No signals
Activation
This breaks anything using one of these domains, and requires migrating to other hostnames.
Security
None
Debuggability
These will act like any other invalid URL. Behavior is context dependent. Since this is logic deep within GURL, and GURLs are created in a great many places, console warnings specifically for this seem not practical.
Is this feature fully tested by web-platform-tests?
No. Javascript URL construction is tested, but URLs are used in a great many other places, which don't have test coverage, since DNS lookups for these domains must succeed in the first place for the tests to be meaningful.Flag name
Requires code in //chrome?
FalseTracking bug
https://crbug.com/1237032Estimated milestones
| DevTrial on desktop | 95 |
| DevTrial on Webview | 95 |
Link to entry on the Chrome Platform Status
https://www.chromestatus.com/feature/5679790780579840
Yoav Weiss
Contact emails
mme...@google.comExplainer
NoneSpecification
https://url.spec.whatwg.org/Summary
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid, and looked up via DNS (e.g., http://foo.127.1/). Per the Public Suffix List spec, the eTLD+1 of the hostname in that URL should be "127.1". If that is ever fed back into a URLs, "http://127.1/" is mapped to "http://127.0.0.1/" by the URL spec, which seems potentially dangerous. "127.0.0.0.1" could also potentially be used to confuse users. We want to reject URLs with these hostnames.
Blink component
Internals>Network>DNSMotivation
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid hostnames, and looked up via DNS. Example hostnames: 127.0.0.0.1, foo.0.1, 10.0.0.09, 08.1.2.3. These can be problematic for the following reason: * "http://foo.127.1/" has an eTLD+1 of "127.1", per the public suffix list spec. If that's ever used as the hostname in a new URL, however, as in "http://127.1", it will then get mapped to "http://127.0.0.1/", per the URL spec, which is a different host, which is not safe. * "http://127.0.0.0.1" and "http://1.2.3.09", both of which are looked up via DNS rather than failing or being treated as IPv4 hostnames, also seem potentially confusing. While no exploit is currently known here, we want to remove support for these as a preventative security measure. The URL spec has been updated so that any URL with a hostname ending in a number that's not an IPv4 address (including, e.g., http://foo.1./, but not http://foo.1../) is considered invalid. Since this is part of the URL spec, not the DNS spec, we want to reject these URLs are the GURL layer, for URLs with appropriate protocols (http, https, ws, wss, file). For consistency, we should also fail DNS lookup attempts of these sorts of hostnames.
Initial public proposal
https://github.com/whatwg/url/pull/619TAG review
Not required for an Intent to Deprecate, I believe.TAG review status
Not applicableRisks
Interoperability and Compatibility
Any URL with an affected hostname will fail to load, and will need to be migrated to another hostname. URLs of this form do appear to be in use, though it's not clear under what circumstances. No entry in the public suffix list is affected. Affected URLs make up no more than 0.0003% of hostnames looked up via the host resolver on any platform, and are basically not used in any file URLs, according to our metrics.
On OSX and Android, about 90% of host resolver lookups for these hostnames succeed, 60% do on Linux, and 2% on Windows and ChromeOS.
To allow for emergency disabling in case of wider than expected breakage, I intend to add a feature for it, and do a 50% field trial on pre-release channels, though plan to just enable the feature, rather than do a gradual rollout to stable, given the low usage.
Gecko: Positive (https://github.com/whatwg/url/pull/619#issuecomment-890826499)
WebKit: In development (https://bugs.webkit.org/show_bug.cgi?id=228826)
Web developers: No signalsActivation
This breaks anything using one of these domains, and requires migrating to other hostnames.
Security
None
Debuggability
These will act like any other invalid URL. Behavior is context dependent. Since this is logic deep within GURL, and GURLs are created in a great many places, console warnings specifically for this seem not practical.
Is this feature fully tested by web-platform-tests?
No. Javascript URL construction is tested, but URLs are used in a great many other places, which don't have test coverage, since DNS lookups for these domains must succeed in the first place for the tests to be meaningful.Flag name
Requires code in //chrome?
FalseTracking bug
https://crbug.com/1237032Estimated milestones
DevTrial on desktop 95 DevTrial on Webview 95 Link to entry on the Chrome Platform Status
https://www.chromestatus.com/feature/5679790780579840This intent message was generated by Chrome Platform Status.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEK7mvq%2Bfnau%3DE%2BONhe0kr9HOpN84eCpoub84%3DswKzPkrGzi5A%40mail.gmail.com.
Yoav Weiss
On Wed, Aug 18, 2021 at 4:53 PM Yoav Weiss <yoav...@chromium.org> wrote:
On Wed, Aug 18, 2021 at 8:47 PM 'Matt Menke' via blink-dev <blin...@chromium.org> wrote:Contact emails
mme...@google.comExplainer
NoneSpecification
https://url.spec.whatwg.org/Summary
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid, and looked up via DNS (e.g., http://foo.127.1/). Per the Public Suffix List spec, the eTLD+1 of the hostname in that URL should be "127.1". If that is ever fed back into a URLs, "http://127.1/" is mapped to "http://127.0.0.1/" by the URL spec, which seems potentially dangerous. "127.0.0.0.1" could also potentially be used to confuse users. We want to reject URLs with these hostnames.
Blink component
Internals>Network>DNSMotivation
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid hostnames, and looked up via DNS. Example hostnames: 127.0.0.0.1, foo.0.1, 10.0.0.09, 08.1.2.3. These can be problematic for the following reason: * "http://foo.127.1/" has an eTLD+1 of "127.1", per the public suffix list spec. If that's ever used as the hostname in a new URL, however, as in "http://127.1", it will then get mapped to "http://127.0.0.1/", per the URL spec, which is a different host, which is not safe. * "http://127.0.0.0.1" and "http://1.2.3.09", both of which are looked up via DNS rather than failing or being treated as IPv4 hostnames, also seem potentially confusing. While no exploit is currently known here, we want to remove support for these as a preventative security measure. The URL spec has been updated so that any URL with a hostname ending in a number that's not an IPv4 address (including, e.g., http://foo.1./, but not http://foo.1../) is considered invalid. Since this is part of the URL spec, not the DNS spec, we want to reject these URLs are the GURL layer, for URLs with appropriate protocols (http, https, ws, wss, file). For consistency, we should also fail DNS lookup attempts of these sorts of hostnames.
Initial public proposal
https://github.com/whatwg/url/pull/619TAG review
Not required for an Intent to Deprecate, I believe.TAG review status
Not applicableRisks
Interoperability and Compatibility
Any URL with an affected hostname will fail to load, and will need to be migrated to another hostname. URLs of this form do appear to be in use, though it's not clear under what circumstances. No entry in the public suffix list is affected. Affected URLs make up no more than 0.0003% of hostnames looked up via the host resolver on any platform, and are basically not used in any file URLs, according to our metrics.
Do we have reason to believe these hostnames are not legitimate ones?
Unfortunately, we have no insight into them - they could be mistyped URLs sent to typo squatting ISPs that OSX lets through but the Windows host resolver blocks, and various flavors of Linux treat differently. Or they could be mapped via a hosts file, or they could be hostnames that only resolve on public networks. Could be some network tool that uses them when installed locally, but is only available on certain platforms. No reason to think one possibility is more likely than the others.
On OSX and Android, about 90% of host resolver lookups for these hostnames succeed, 60% do on Linux, and 2% on Windows and ChromeOS.
Do you know where those failures are coming from?
Could be typos, could be the Windows and ChromeOS host resolvers don't let them through. Since we've had no filed bugs about them, I suspect the failures are not deliberate navigations or intended network requests. I'm much more interested in where the successes are coming from, myself.
To allow for emergency disabling in case of wider than expected breakage, I intend to add a feature for it, and do a 50% field trial on pre-release channels, though plan to just enable the feature, rather than do a gradual rollout to stable, given the low usage.
Gecko: Positive (https://github.com/whatwg/url/pull/619#issuecomment-890826499)Can you file an official position request? https://bit.ly/blink-signals
Done for Mozilla: https://github.com/mozilla/standards-positions/issues/568Should I also do this for WebKit as well? They have in process CLs, so not sure if it's still needed.
Yoav Weiss
We have UKM on their number (0.0003% of DNS lookups on OSX, less elsewhere - we can't meaningfully instrument percent of created GURLs), but we don't have their hostnames, what they resolve to, or know anything else about them, unfortunately.Navigation to a subset of these as frame URLs were broken at one point - I'm pretty sure the breakage even made it to stable: https://crbug.com/1173238. There were no reports of problems. Only non-IPv4 URLs where the last two components were broken, though, and it didn't affect subresources. On OSX and Android, over 99% of successfully resolved problematic hostnames fit into that bucket, though on Linux, only about 2% do.That doesn't give us any hard conclusions, except they're either not deliberate navigations on OSX/Android, or they're not navigations.
Yoav Weiss
I created the title using Chrome Status's deprecation template, so any confusion should be blamed on that.
I used the "Draft Intent to Deprecate and Remove email" button, and assume I'd need to do a "Draft Intent to Ship email" before shipping to stable, after a 50% trial on prerelease channels.
Yoav Weiss
Department of odd facts:- The ICANN rules for new TLDs forbid all top level domain names that start with a digit- The IDNA rules for bidirectional scripts forbid domain names that start with a digit (Unicode bidi afficandoes will know why)- The only real reason why leading digits aren't outlawed in domain names at the second level is 3com.It seems safe to say that no legitimate fully qualified hostname will ever have a last component consisting only of digits.
That means the only time we could get a legitimate hostname is for something that has to be resolved via a search path.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWB4wVuGshgPaLVXp%3DYsWUiXgJhUABD3ZFJ9xbhg1J3ww%40mail.gmail.com.
Harald Alvestrand
That means the only time we could get a legitimate hostname is for something that has to be resolved via a search path.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWB4wVuGshgPaLVXp%3DYsWUiXgJhUABD3ZFJ9xbhg1J3ww%40mail.gmail.com.
Harald Alvestrand
Matt Menke
On Wed, Aug 18, 2021 at 8:47 PM 'Matt Menke' via blink-dev <blin...@chromium.org> wrote:Contact emails
mme...@google.comExplainer
NoneSpecification
https://url.spec.whatwg.org/Summary
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid, and looked up via DNS (e.g., http://foo.127.1/). Per the Public Suffix List spec, the eTLD+1 of the hostname in that URL should be "127.1". If that is ever fed back into a URLs, "http://127.1/" is mapped to "http://127.0.0.1/" by the URL spec, which seems potentially dangerous. "127.0.0.0.1" could also potentially be used to confuse users. We want to reject URLs with these hostnames.
Blink component
Internals>Network>DNSMotivation
Most hostnames that aren't valid IPv4 addresses, but end in numbers are treated as valid hostnames, and looked up via DNS. Example hostnames: 127.0.0.0.1, foo.0.1, 10.0.0.09, 08.1.2.3. These can be problematic for the following reason: * "http://foo.127.1/" has an eTLD+1 of "127.1", per the public suffix list spec. If that's ever used as the hostname in a new URL, however, as in "http://127.1", it will then get mapped to "http://127.0.0.1/", per the URL spec, which is a different host, which is not safe. * "http://127.0.0.0.1" and "http://1.2.3.09", both of which are looked up via DNS rather than failing or being treated as IPv4 hostnames, also seem potentially confusing. While no exploit is currently known here, we want to remove support for these as a preventative security measure. The URL spec has been updated so that any URL with a hostname ending in a number that's not an IPv4 address (including, e.g., http://foo.1./, but not http://foo.1../) is considered invalid. Since this is part of the URL spec, not the DNS spec, we want to reject these URLs are the GURL layer, for URLs with appropriate protocols (http, https, ws, wss, file). For consistency, we should also fail DNS lookup attempts of these sorts of hostnames.
Initial public proposal
https://github.com/whatwg/url/pull/619TAG review
Not required for an Intent to Deprecate, I believe.TAG review status
Not applicableRisks
Interoperability and Compatibility
Any URL with an affected hostname will fail to load, and will need to be migrated to another hostname. URLs of this form do appear to be in use, though it's not clear under what circumstances. No entry in the public suffix list is affected. Affected URLs make up no more than 0.0003% of hostnames looked up via the host resolver on any platform, and are basically not used in any file URLs, according to our metrics.
Do we have reason to believe these hostnames are not legitimate ones?
On OSX and Android, about 90% of host resolver lookups for these hostnames succeed, 60% do on Linux, and 2% on Windows and ChromeOS.
Do you know where those failures are coming from?
To allow for emergency disabling in case of wider than expected breakage, I intend to add a feature for it, and do a 50% field trial on pre-release channels, though plan to just enable the feature, rather than do a gradual rollout to stable, given the low usage.
Gecko: Positive (https://github.com/whatwg/url/pull/619#issuecomment-890826499)
Can you file an official position request? https://bit.ly/blink-signals
Matt Menke
Matt Menke
Matt Menke
On OSX and Android, about 90% of host resolver lookups for these hostnames succeed, 60% do on Linux, and 2% on Windows and ChromeOS.
Harald Alvestrand
Matt Menke
Yoav Weiss
Matt Menke
Matt Menke
Yoav Weiss
Sorry, that should be "before instantiating content" (Or "before calling ContentMain")On Sun, Aug 22, 2021 at 4:51 PM Matt Menke <mme...@google.com> wrote:I've poked around here, and I'm not sure it's possible to wire this up to a base::Feature. Headless runs some code before instantiating context, including parsing a remote debugging IP address from the command line. This results in calling the IPv4 parser before content, and thus the global feature list, is loaded, which results in the feature code CHECKing. Even if we figured out a way to work around this, I'm concerned that other code not exercised by the trybots may be doing the same thing, which could result in a lot of unexpected surprises.I guess we could add a command line switch to bypass the check, though that's more difficult to use than a base::Feature, and can't be set server-side. Open to other ideas.
Matt Menke
I've poked around here, and I'm not sure it's possible to wire this up to a base::Feature. Headless runs some code before instantiating context, including parsing a remote debugging IP address from the command line. This results in calling the IPv4 parser before content, and thus the global feature list, is loaded, which results in the feature code CHECKing. Even if we figured out a way to work around this, I'm concerned that other code not exercised by the trybots may be doing the same thing, which could result in a lot of unexpected surprises.I guess we could add a command line switch to bypass the check, though that's more difficult to use than a base::Feature, and can't be set server-side. Open to other ideas.
Matt Menke
Yoav Weiss
Is one LGTM enough for intent-to-deprecates, or do I need 3?
Yoav Weiss
Matt Menke
Matt Menke
Chris Harrelson
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/cff2a72a-f37b-4cc9-aeff-51ea86c47674n%40chromium.org.
Mike West
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.