Intent to Ship: Support non-special scheme URLs

[Hayato Ito]

Contact emails

hay...@chromium.org

Explainer

http://bit.ly/url-non-special

Specification

https://url.spec.whatwg.org/

Summary

Support non-special scheme URLs.

Previously, Chromium's URL parser didn't handle non-special scheme URLs properly. It treated these URLs as “opaque paths”, which didn’t align with the URL Standard.

Now, Chromium’s URL parser correctly processes non-special URLs.

Examples:

Before:

> const url = new URL("git://host/path");

> url.host

""

> url.pathname

"//host/path"

> url.host = "newhost";

> url.host

""

> const url = new URL("git://a b/path");

> url.pathname

"//a b/path"

After:

> const url = new URL("git://host/path");

> url.host

"host"

> url.pathname

"/path"

> url.host = "newhost";

> url.host

"newhost"

> url.href

"git://newhost/path"

> const url = new URL("git://a b/path");

=> throws Exception. // A space character is not allowed as a hostname.

See http://bit.ly/url-non-special for more details.

As part of our Interop 2024 efforts, this change delivers the following improvements:

• Boosts WPT URL Score: 936 previously failing subtests in the WPT URL tests (link) now pass, raising the score from 87.0% to 94.7%.

• Fixes code relying on incorrect URL behavior: 527 tests  (link) and related code in Chromium that depended on the previous behavior are now fixed or mitigated, including:

• Web tests that relied on non-compliant non-special URL behavior (e.g. “javascript://a b” URL)

• Non-special schemes used internally by Chromium code base, including ChromeOS (e.g. “steam:”, “materialized-view://”, “cros-apps://”)

Blink component

Internals>Network

TAG review

Not applicable

Risks

Interoperability and Compatibility

Since Safari and Firefox already support non-special scheme URLs, the likelihood of public websites breaking due to this change is likely low. See here for a rough estimation of the non-special scheme URL usages.

Gecko: Shipped

WebKit: Shipped

Web developers:  Generally seems positive.

Some signals (from interop 2024 discussions) are:

• > Confusion because URL parsers across Blink, Gecko, WebKit, Node, and Deno do not interop well. The root cause is nearly always parser bugs in Blink or Gecko: https://twitter.com/oleg008/status/1699087223751073883

• > URL is very widely used - custom schemes are commonly used for links to native apps, or when dealing with developer tooling like databases. They may also become exceedingly more common with import maps.

Other potential risks and assessments:

• Enterprise usage: It's difficult to predict how non-special URLs are used in the wild, especially by enterprise customers with in-house apps. While adding an Enterprise Policy was considered to mitigate risks, technical limitations make it difficult to support URLs. See  http://bit.ly/url-non-special for more info. We'll disable the feature with Finch (StandardCompliantNonSpecialSchemeURLParsing flag) in case this causes serious issues.

• Impacts on well-known non-special schemes: See here for the impacts on “javascript://”, “data:”, and so on.

• Impacts on dependent components: This change affects components relying on URL behavior, like Origin. See the Security section below.

Security

In Chromium, GURL, KURL, and web-facing URL APIs share the common URL parser backends, which reside in //url. As a result, this web-facing change will also affect core components like url::Origin, kurl::SecurityOrigin.

For detailed information on how url::Origin, kurl::SecurityOrigin, and web-facing url.origin are impacted, please refer to this CL’s description.

TL;DR. This is a complex issue due to historical reasons. While most components remain unaffected, there are some nuances, particularly regarding the “Android WebView Hack”. We’ve preserved the current Origin behavior for Android WebView.

WebView application risks

Beyond the aforementioned "Android WebView Hack", there are no other changes specific to WebView.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Is this feature fully tested by web-platform-tests?

Yes (dashboard)

Flag name

StandardCompliantNonSpecialSchemeURLParsing

Requires code in //chrome?

False

Tracking bug

https://crbug.com/1416006

Estimated milestones

M130

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5201116810182656

Links to previous Intent discussions

Previous I2S. The previous I2S mail was sent last year but please consider this intent to ship as a new one.

--

Hayato

[Philip Jägenstedt]

LGTM1, very happy to see this make progress!

It's unfortunate that rolling this out with Finch and enterprise policy does not look feasible, but very good that we still have a kill switch. Keep a close eye on incoming bugs will also be important.

Good luck! :D

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFpjS_1_R%3D%2BHXYgTCuLD_WGR0foLKVnxAU9am1QbHyAZ%3D%2B3Ohw%40mail.gmail.com.

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYdo4SEr05zmLKj4U4OD3f8Gx0taa-7B%3D-5n1K69b6_mgw%40mail.gmail.com.

[Chris Harrelson]

OOps, I take back my LGTM. Please fill out the cross-functional reviews:

[Hayato Ito]

Done. I filled the cross-functional reviews (Privacy, Security, Enterprise, Debuggability (N/A), and Testing). Thanks!

Note: The platform status has two "Prepare to ship" entries. Please ignore the second one. It seems I can't delete it.

--

Hayato

[Yoav Weiss (@Shopify)]

Thanks for working on this!!

On Thu, Aug 8, 2024 at 10:09 AM Hayato Ito <hay...@chromium.org> wrote:

Any particular section in that doc that talks through or estimates the compat risk?

 

--

[Hayato Ito]

I've added a new section, "Compatibility Risks", to the doc, organizing the content more clearly.

Thanks!

--

Hayato

[Domenic Denicola]

I'm very excited to see this work proceeding and am looking forward to being able to LGTM. A few requests before we get there:

• From what I understand we still won't be spec-compliant for android:, drivefs:, steam:, chromeos-steam:, and materialized-view:. Can you add failing web platform tests for all of those cases, to ensure that we capture this non-compliance?
• I don't understand the role of the scheme registry after we ship this change. What will it do? The doc says "(TBD) Remove the Scheme Registry. This can be yet another non-trivial project. This document does not cover this task." Are there still web-exposed behaviors between non-registered and registered schemes? Is there a danger that people will add more schemes to the registry and cause URL parsing to change? Or is this purely about code cleanup as after this change the scheme registry is a no-op?
  
• In general it would be great to have a section of the doc covering future work, such as fixing those exceptional schemes, removing the scheme registry, and anything else that you have in mind. In particular I'm interested in which of these future work projects have compat impacts so we can know how web developers might see URL parsing change in the future.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFpjS_1%3D48YmdprXVdy_3SWRqneFFcy6BRuaw_w%2BxvrjyAc2CA%40mail.gmail.com.

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAM0wra9m1qFkKLjKPu3B-fQMtvf0acXo3YGhZTWGeQcAgh-MVg%40mail.gmail.com.

[Merih Akar]

Hi,

In our app we're using a custom protocol, and I'm trying to understand if this change would affect how we handle some URL schemes we use.

In the interoperability section you mentioned that firefox already supports non-special scheme URLs, however when I test right now the result in firefox is the same as old behavior, not like the example you shared as "after":

I tried to test the new behavior in chrome canary, but the result was the same there too, so is there any place where we can test parser changes to see if it breaks our code? Or is this change not visible to web authors?

Thanks,

--

merih

[Hayato Ito]

Thanks for the feedback!

On Fri, Aug 9, 2024 at 5:19 PM Domenic Denicola <dom...@chromium.org> wrote:

I'm very excited to see this work proceeding and am looking forward to being able to LGTM. A few requests before we get there:

• From what I understand we still won't be spec-compliant for android:, drivefs:, steam:, chromeos-steam:, and materialized-view:. Can you add failing web platform tests for all of those cases, to ensure that we capture this non-compliance?

Sounds good!  I'm adding tests here: https://crrev.com/c/5790445.
I'll make sure to land this CL before enabling the flag.

 

• I don't understand the role of the scheme registry after we ship this change. What will it do? The doc says "(TBD) Remove the Scheme Registry. This can be yet another non-trivial project. This document does not cover this task." Are there still web-exposed behaviors between non-registered and registered schemes? Is there a danger that people will add more schemes to the registry and cause URL parsing to change? Or is this purely about code cleanup as after this change the scheme registry is a no-op?
  
• In general it would be great to have a section of the doc covering future work, such as fixing those exceptional schemes, removing the scheme registry, and anything else that you have in mind. In particular I'm interested in which of these future work projects have compat impacts so we can know how web developers might see URL parsing change in the future.

Re the 2nd and the 3rd points, I've added a "Future Work" section to the document to capture what I have in mind and what should be done next. Thanks for the suggestion!
I've also removed some outdated content from the document like "(TBD) Remove the Scheme Registry".

--

Hayato

[Hayato Ito]

Thanks for checking it. I confirmed that Firefox ESR (15.13) behaves the same way, as you confirmed. That's definitely not what I expected. My bad.

I looked for a specific bug in Mozilla's Bugzilla. It seems there isn't one for this, however, there are some bugs related to non-special schemes like "non-spec:" in the Interop 2024 meta bug.
(Go https://bugzilla.mozilla.org/show_bug.cgi?id=1876105 > Details > tree)

Given that, I've updated the platform status for Firefox from "Shipped" to "Positive".
The previous I2S also had "Firefox: Positive", which I might have misinterpreted as "Shipped". Sorry about the lack of confirmation.

Regarding Chrome, the flag is not yet enabled even in Canary. Thus, you need the command line argument, `--enable-features=StandardCompliantNonSpecialSchemeURLParsing`, to enable the flag in Chrome at this point.

--

Hayato

[Domenic Denicola]

Excellent, thank you!

Your updated section of the document states

> There have been several non-special schemes (like "isolated-app://") which are registered in the SchemeRegistry so that they are parsed if they were special scheme URLs.

I believe this means that the following schemes:

• isolated-app:
• chrome-native:
• chrome-search:
• chrome-distiller:
• android-app:
• chrome-resource:
• chrome-extension:
• fuchsia-dir:

will not match the URL Standard either. (Testing with --enable-features=StandardCompliantNonSpecialSchemeURLParsing on the live URL viewer seems to confirm this.)

In that case, please add those schemes to your CL as well. Then I am happy to approve!

[Hayato Ito]

Thanks! That makes sense too. I'm adding those schemes to the CL.

--

Hayato

[Domenic Denicola]

Excellent, thank you! LGTM3, so excited to see this roll out.