Intent to Ship: Support URLs with non-special schemes

[Jiacheng Guo]

Contact emails

g...@google.com

Explainer

None

Specification

https://url.spec.whatwg.org/#url-parsing

Summary

URLs with non-special schemes will be supported in chrome. `non-speicial://test.com:1234/path` will be become a valid URL. One can access and set the URL properties such as host, port and path via the URL class.

Blink component

Blink>JavaScript>API

TAG review

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Positive

WebKit: Positive

Web developers: No signals

Other signals:

Ergonomics

No significant risks.

Activation

No significant risks.

Security

data:// and javascript:// URLs handling is not modified due to their critical role.

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Debuggability

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Flag name

NonSpeicalSchemeURLParsing

Requires code in //chrome?

False

Tracking bug

https://crbug.com/1416006

Sample links

https://chromium-review.googlesource.com/c/chromium/src/+/4273893

Estimated milestones

No milestones specified

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5201116810182656

This intent message was generated by Chrome Platform Status.

[Mike Taylor]

Do URLs with an intent:// scheme have any security considerations, or implications for WebView? (I don't know, hopefully someone who does can answer. :))

Debuggability

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Flag name

NonSpeicalSchemeURLParsing

Requires code in //chrome?

False

Tracking bug

https://crbug.com/1416006

Sample links

https://chromium-review.googlesource.com/c/chromium/src/+/4273893

Estimated milestones

No milestones specified

Anticipated spec changes

Open questions about a feature may be a source of future web compat or interop issues. Please list open issues (e.g. links to known github issues in the project for the feature specification) whose resolution may introduce web compat/interop risk (e.g., changing to naming or structure of the API in a non-backward-compatible way).

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5201116810182656

This intent message was generated by Chrome Platform Status.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJQw1Nzk847XL759vMSQaF3L5zvtykg6UfQvuss4diyU-h1%3Duw%40mail.gmail.com.

[Yoav Weiss]

On Wed, Feb 22, 2023 at 4:43 PM Mike Taylor <mike...@chromium.org> wrote:

On 2/22/23 8:21 AM, 'Jiacheng Guo' via blink-dev wrote:

Contact emails

g...@google.com

Explainer

None

An explainer (even inline) would be helpful to get a better understanding of what this change does.

Does it impact only URL() object construction? What is happening today? What will happen after this change lands?  

Specification

https://url.spec.whatwg.org/#url-parsing

Summary

URLs with non-special schemes will be supported in chrome. `non-speicial://test.com:1234/path` will be become a valid URL. One can access and set the URL properties such as host, port and path via the URL class.

Blink component

Blink>JavaScript>API

TAG review

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Positive

WebKit: Positive

Any links to those positive signals?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7cdf2693-c8a3-d263-0eb0-a44a2390979e%40chromium.org.

[Harald Alvestrand]

Is there a blacklist of "special schemes" that this change won't touch? Who maintains that list?

This seems a bit dangerous, in that if a new scheme is deployed that is "special", code intended for handling non-special schemes will try to parse it.

Note that the term "special" in the URL specification (https://url.spec.whatwg.org/#special-scheme) refers strictly to ftp, file, http, https, ws and wss; there's nothing "special" about urn, turn, stun or any of the other standardized schemes that don't use the // syntax.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfVfGhV%2BDRzpCjGFoHg7EXb325nHz3nu4OSQVTTC6bkS1A%40mail.gmail.com.

[Mike Taylor]

Hi Jiacheng,

Friendly ping on Harald's and my questions. :)

thanks,
Mike

[Jiacheng Guo]

Sorry for the late reply,

I've created a doc on the security concerns for non-special URLs. The general idea is to support non-special URLs and add a blocklist where the URLs can only have opaque hosts.

I added the security team to ask for their comments as well.

Jiacheng Guo

[Mike Taylor]

Thanks for the doc - if "WPT URL failure triage" is what you intended to send, could you point out which section contains the security concerns? (Or maybe just linked the wrong doc on accident?)

[Jiacheng Guo]

Sorry I sent the wrong document

It should be https://docs.google.com/document/d/1edoInUnxwJAGN0264yFRvs6Yi5ptb37HvFYkBNnz2YQ/edit?usp=sharing

[Domenic Denicola]

Hi Jiacheng,

Thanks again for all this interop work!

I don't think I understood the process that led to special treatment for data:, javascript:, intent:, urn:, turn:, and stun:. It seems like the intent is to not follow the standard precisely for those schemes, right? I guess that might be reasonable as a stepping stone, but I want to make sure we're tracking our failure to follow the standard there, and hopefully eventually fixing it.

I've filed https://github.com/web-platform-tests/wpt/issues/38970 to discuss adding more test coverage. To help us with that, can you provide an example of how the blocklist your document discusses will work? That is, the document says 

> Add a blocklist for non-special schemes. The schemes in the block list must have an opaque host.

Since there's no such list in the URL Standard itself, I'm assuming this means those schemes will have nonstandard behavior. But I don't understand what nonstandard behavior is implied by "must have an opaque host". Can you give an example of, e.g., a stun: URL, which will parse differently in the URL Standard vs. Blink's implementation, due to this blocklist?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJQw1NwdWUn7OOrEgGjGeZV%3DHa_niTT0Jg_yv7j7uN2uRL7fcA%40mail.gmail.com.

[Jiacheng Guo]

Currently blink disallows non-special URLs with a host such as about://example.com/ or stun://test.com:8080/. The allowed URLs can be about:example or stun:test.com.

The main concern for implementing spec compliant parsing of the URLs is we do not know whether other chrome components assume opaque hosts for these URLs. We wonder if there will be potential issues in the URL handling.

[Domenic Denicola]

Hmm, I'm not sure that answered my question. But let me try guessing at an answer:

An example of a URL that will still parse differently after this change, is stun://test.com:8080/. That will parse as pathname = "//test.com:8080/" in Chromium, even after this change, whereas per the standard, that should parse as port = 8080, hostname = "test.com", pathname = "/".

Is that correct? If so, I'll be sure we add failing web platform tests for cases like that, so that we don't inadvertently get full credit for fixing our non-special URL parsing code when we haven't finished that work yet.

[Jiacheng Guo]

Yes, the behavior for the schemes in the blocklist will not change before and after the change.

[Domenic Denicola]

Great, thanks! I've added a variety of tests for these cases in https://github.com/web-platform-tests/wpt/pull/38972 , to better track our future work toward full spec compliance. In the meantime, any incremental step toward interop is an improvement, so I want to reiterate how happy I am that you're working on this!

[Charlie Reis]

[-CSA, +security-dev]

On Fri, Mar 10, 2023 at 3:31 AM 'Jiacheng Guo' via blink-dev <blin...@chromium.org> wrote:

I added the security team to ask for their comments as well.

Thanks, but moving to the external security-dev@ list instead of the internal CSA team list.  I think Blink Intents already get reviewed for security, and antoniosartori@ might be looking at this one?

I do think we would want to be very careful about allowing schemes to start supporting hosts when they didn't before.  I expect that would cause a lot of confusion for data: and javascript:, so I'm glad they're on the blocklist for supporting hosts going forward.  What schemes are not ending up on the blocklist and would change behavior?  Would something like content: start supporting schemes, and would that cause problems?

Also, would this proposed change mean that any new schemes would be possible to request or navigate to, or just that they could be parsed?

Charlie

You received this message because you are subscribed to the Google Groups "Chrome Security Architecture Core team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chrome-security-archit...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/chrome-security-architecture-core/CAM0wra_zfah%3DBsGL_GXW_RY7CtFvY646yoKvRiFGosTTL9FxjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/a/google.com/d/optout.

[Jiacheng Guo]

Thanks for the review!

The proposed change would just be parsed. The navigation behavior will remain the same.

Jiacheng Guo

[Antonio Sartori]

We had a look at this in the Web Platform Security and Privacy reviews. Thanks for clarifying that the proposed change is just for parsing. Then I believe that this is fine in principle, but I am not totally sure about the actual changes needed in the code: I just fear we might have some piece of the codebase which assumes that it will never get as input a non-special URL (since it used to be invalid), and changing that might become a security issue? Could you explain in more detail what you are going to change? Maybe you even have a draft CL?

[Torne (Richard Coles)]

Hi folks,

Will this change potentially impact WebView? WebView currently allows arbitrary URL schemes to be used, as well as a number of Android-specific schemes: https://source.chromium.org/chromium/chromium/src/+/main:android_webview/common/aw_content_client.cc;l=39;drc=e672a665ffa8fe4901184f03922e2cc548399da5

I'm not super clear on what the actual effect of the change will be here, but currently WebView apps might depend on the fact that if they load "foo://bar" this is treated as origin "foo://" and is same-origin with "foo://baz". There may be other ways this is relevant as well; I'm unfortunately not certain what all the implications of WebView's support for custom schemes actually are.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOzWxF6wJ17ChhQbaDGz4O8X-zjo0dx7tS-_zuXzeOUKO%2BynPQ%40mail.gmail.com.

[Jiacheng Guo]

The Draft CL can be found here: https://chromium-review.googlesource.com/c/chromium/src/+/4273893

[Jiacheng Guo]

As to the Android WebView hack, currently "foo://test" on Android Webview has a valid security origin with an empty host (foo://).

By the standard the origin will be "foo://test". After the patch the behavior will be affected.

The hack traces back to crbug.com/896059 and b/117514441. For some reason, Gmail depends on this behavior to work.

Chances are that we need to keep that hack.

[Torne (Richard Coles)]

On Thu, 16 Mar 2023 at 06:16, Jiacheng Guo <g...@google.com> wrote:

As to the Android WebView hack, currently "foo://test" on Android Webview has a valid security origin with an empty host (foo://).

By the standard the origin will be "foo://test". After the patch the behavior will be affected.

OK - so yeah, this is going to be a breaking change for *some* number of apps that use WebView with arbitrary URL schemes.

 

The hack traces back to crbug.com/896059 and b/117514441. For some reason, Gmail depends on this behavior to work.

Chances are that we need to keep that hack.

Gmail is not the only application that depends on this; quite a few apps use their own arbitrary URL schemes to load content and if the origin changes then the app may be affected. So, yes, we either need to keep the current behaviour in WebView as-is, or at least spend some time gathering data to try to understand the impact of changing it.

From talking to app developers it seems like one of the reasons this is common is because iOS's WKWebView does not allow the use of HTTP/HTTPS URLs to load custom local content, and so apps/frameworks that want to support both iOS and Android are likely to use custom schemes to keep the behaviour as similar as possible between platforms.

[Jiacheng Guo]

Hi team,

I've conducted some statistics on the WebArchive HTTP responses for the most frequently used non-speical schemes.

The query goes over ten million non-sepcial schemes in the latest WebArchive HTTP responses and counts for the unique non-special schemes.

The result can be found here:

https://docs.google.com/spreadsheets/d/1Hppmix1F6vW2iaTybIghhQ-aO8CxjE4DvoUvOMOu144/edit?usp=sharing

I'd like to ask the security folks to review the most frequently used schemes and list the schemes that may raise security concerns.

Much Thanks!

Jiacheng Guo

[Antonio Sartori]

Thanks for providing more information!

I had a look at your CL (https://crrev.com/c/4273893) and I commented there, but I don't have the impression it does all you want. In particular, I think setting host and port for non-special schemes is still not allowed.

That aside, it would be interesting to understand what else is supposed to change. For example, at the moment IIUC non-standard schemes always have opaque origins (apart from "the WebView hack"). If that changes, we might end up with URLs which were cross-origin before and now are same-origin. That might have security implications, which I am unsure about.

As for the non-special schemes in the list, I don't think they should really matter unless they are understood and processed by chrome, and I don't think any of them are, but I don't feel I am  particularly expert on that so I would let creis@ and others comment on this.

Antonio

[Jiacheng Guo]

I'd like to determine the design for supporting the non-special URLs in the thread and update it to the CL.

A possible while conservative way is to force an opaque host for any schemes registered by the browser and keep the Webview hack. Maybe we can start from this approach and gradually remove the registered hosts?

[Charlie Reis]

I'm not sure that I'm going to be of much help in this discussion, since I don't know about all the various scheme dependencies in Chrome, and I'm quite backlogged.  I'm mainly concerned about breaking any assumptions about cases where a URL's origin is inherited but not parsed from the URL (e.g., about:blank) or intentionally opaque (e.g., data:).  I also see the CL is proposing changes to url/ itself, which means it could affect all uses of URLs in Chrome (including navigation) and not just how they are parsed within JavaScript, so a mistake here could be pretty significant.  I don't have a good enough understanding of the change to say whether it's going to cause problems or not, though.

I'll loop in @Łukasz Anforowicz as the author of https://chromium.googlesource.com/chromium/src/+/main/docs/security/origin-vs-url.md and @Daniel Cheng for knowledge about GURL and perhaps some of the Blink changes being considered, in case they're able to help spot concerns or identify that the proposal is safe.

(As a side note, I do support interop work to make the standard and browser behavior safely agree, and I just don't know if this case requires changes to Chrome, the standard, or both.)

Charlie