Intent to Deprecate and Freeze: The User-Agent string

[Yoav Weiss]

Contact emails

yoav...@chromium.org, aaro...@chromium.org

Summary

We want to freeze and unify (but not remove) the User Agent string in HTTP requests as well as in `navigator.userAgent`

Motivation

The User-Agent string is an abundant source of passive fingerprinting information about our users. It contains many details about the user’s browser and device as well as many lies ("Mozilla/5.0", anyone?) that were or are needed for compatibility purposes, as servers grew reliant on bad User Agent sniffing.

On top of those privacy issues, User-Agent sniffing is an abundant source of compatibility issues, in particular for minority browsers, resulting in browsers lying about themselves (generally or to specific sites), and sites (including Google properties) being broken in some browsers for no good reason.

The above abuse makes it desirable to freeze the UA string and replace it with a better mechanism. There have been past attempts at UA string freezing from the Safari team, but without an alternative way to perform UA based content-negotiation, they had to be partially reverted.

 

The User Agent Client Hints (UA-CH) feature provides an alternate source for the information the User-Agent string provides, both in its request header form as well as its JS API one. 

Its main advantages are:

• It provides the required information only when the server requests it, over secure connections, making any fingerprinting that relies on it be active fingerprinting, which enables such use to be audited, as well as acted-upon by the browser (e.g. in a future implementation of the Privacy Budget).

• It provides the information in small increments, so servers are only exposed to the information they need and request, rather than being exposed to the full gamut of the UA string even if they are just trying to figure out one detail about the browser. (e.g. brand and major version)

• Since it provides the information via dedicated fields, it enables better ergonomics and makes it less likely for servers to get it wrong and cause compatibility issues.

• And finally, starting fresh will enable us to drop a lot of the legacy baggage that the UA string carries (“Mozilla/5.0”, “like Gecko”, “like KHTML”, etc) going forward.

Once UA-CH ships as an alternative means for browser-specific content adaptation, we would like to freeze the User-Agent string. 

We propose to deprecate at M81 (starting to emit console warnings in pages that read that string in JS), freeze its version information at M83, and unify strings of different devices at M85. See detailed freezing plan below. 

This timeline provides 3 months for developers to move to the new mechanism for their future browser and OS version needs, and 6 months for more sophisticated OS or device specific targeting.

Freezing plan

Different parts of the UA string have different compatibility implications. 

Some parts of it, such as the browser version and the OS version, can be frozen without any backwards compatibility implications. Values that worked in the past will continue to work in the future.

Other parts, such as the model (for mobile devices) and the OS platform, can have implications on sites that tailor their UI to the underlying OS or that target a very specific model in their layout. Such sites will need to migrate to use UA-CH.

As such we are planning to freeze the parts that are amenable to freezing fairly early, and gradually unify the rest.

Milestone	Stable date	Action
M81	Mid March ‘20	Deprecate access to `navigator.userAgent`
M83	Early June ‘20	Freeze browser version and unify OS versions
M85	Mid September ‘20	Unify desktop OS string as a common value for desktop browsers. Unify mobile OS/device strings as a similarly common value for those at M85 (*)

(*) For the mobile value, we may split it further based on common device dimensions, as a one-time exercise, to reduce the compatibility risk of unification.

Interoperability and Compatibility Risk

The compatibility risk varies at different stages.

For the freezing planned for M83, the compatibility risk is low. Existing UA sniffing code will continue to work as expected. It is only future UA sniffing code that will need to change and use the UA client hints instead.

For the unification planned for M85, the compatibility risk is medium. Some sites can modify their responses based on the OS and device model, and those sites will have to change their UA sniffing code to use UA-CH. We expect such sites to be well maintained (otherwise, how can they keep up with OS UI and device model changes?). Therefore, having 4 releases to modify their code seems sufficient.

In the long term, we expect this change to improve compatibility, as UA sniffing based on UA-CH is bound to be more reliable than the current status quo.

As for interoperability, we have other vendors on board with UA freezing, but not necessarily with the UA Client Hints mechanism, that is supposed to replace it. That can create a tricky situation, where developers would need to rely on the User-Agent string for some browsers and on UA-CH for others.

Edge: Public support

Firefox: Public support for freezing the UA string - “freezing the User Agent string without any client hints—seems worth-prototyping”

Safari: Shipped to some extent. Safari has attempted to completely freeze the UA string in the past, without providing an alternative mechanism. That got a lot of pushback, which resulted in somewhat reverting that decision. Nowadays, their UA string seems frozen, other than updates to the OS version and the browser major version.

Alternative implementation suggestion for web developers

For many (most?) uses of UA sniffing today, a better tool for the job would be to use feature detection. Where feature detection fails developers, UA Client Hints are the right path forward.

Potential deployment hurdles compared to status quo:

• Third party services that rely on the UA string would need to convince the sites that include them to delegate that information to them using Feature Policy. 

• The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay. 

Usage information from UseCounter

If we were going to remove the User-Agent string outright, the answer to that would’ve been “a lot!!!!111”.

Given that we are planning to freeze it, we expect the “removal” to be mostly backwards compatible, barring cases of specific OS or device adaptation.

Client-side UA sniffing use counters show it’s being used by ~90% of sites. But again, since we’re talking about freezing, it should not break most uses.

Entry on the feature dashboard

https://www.chromestatus.com/feature/5704553745874944

[Alex Russell]

Hey Yoav, Aaron:

Glad to see the CH-UA intent to out to ensure that similar information is provided no matter where this intent lands. Can you confirm that, if given approval for this, you will not take action to change UA behavior until CH-UA is shipped?

Presumably, in the context of privacy budgets, a model has been developed that discusses the bits of entropy reduction by default that are gained and the performance tradeoffs of requiring Accept-CH redirectors on nearly all top-level entrypoints for major services that wish to avoid overpolyfilling. Can you publish that analysis?

What work has gone into discussing this change with latency-sensitive properties and services? I'm aware that other UAs have implemented similar changes, but they have also been UAs that tend to benefit from high-end devices and networks by dint of their users being wealthy. If this change had been made 2 years ago, what would the polyfill bloat impact be on our user-base for those serving optimally (e.g., using polyfill.io)?

Regards

[Yoav Weiss]

On Tue, Jan 14, 2020 at 4:25 PM 'Alex Russell' via blink-dev <blin...@chromium.org> wrote:

Hey Yoav, Aaron:

Glad to see the CH-UA intent to out to ensure that similar information is provided no matter where this intent lands. Can you confirm that, if given approval for this, you will not take action to change UA behavior until CH-UA is shipped?

This change is indeed dependent on UA-CH shipping before the first freezing milestone.

 

Presumably, in the context of privacy budgets, a model has been developed that discusses the bits of entropy reduction by default that are gained and the performance tradeoffs of requiring Accept-CH redirectors on nearly all top-level entrypoints for major services that wish to avoid overpolyfilling. Can you publish that analysis?

The assumption is that polyfill avoidance (and most other use-cases for UA string content negotiation) can get by with `Sec-CH-UA` values of brand and major version that will continue to be sent to servers by default.

Is that not the case? If not, can you elaborate on the need to polyfill based on minor versions?

What work has gone into discussing this change with latency-sensitive properties and services? I'm aware that other UAs have implemented similar changes, but they have also been UAs that tend to benefit from high-end devices and networks by dint of their users being wealthy. If this change had been made 2 years ago, what would the polyfill bloat impact be on our user-base for those serving optimally (e.g., using polyfill.io)?

Again, the assumption is that the major version is sufficient for polyfilling.

Even if we assume that it is somehow insufficient, then when polyfill.io is integrated on the client side (e.g. as a `<script>` tag), the requesting site can opt-in for receiving the minor version as well (using `Accept-CH`).

For server side integration, the very-first hit will indeed lack that information at the moment.

We've been discussing various designs to resolve that problem (e.g. by providing the opt-in through lower layers), and hope to solve it in the future. However, we didn't consider this blocking for going ahead with this change.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbfd9b0b-97a4-4775-b4e2-a4da9f87660f%40chromium.org.

[Aaron Tagliaboschi]

On Tue, Jan 14, 2020 at 11:05 AM <josh.d...@realtruck.com> wrote:

a) If the server doesn't get the info on first hit isn't there a performance penalty in that it has to respond with what it wants, then wait for the response just to give the optimal experience?

 

This is indeed something we're looking at, both in terms of handling in the browser and lower levels. It should also be noted that this only happens on navigation requests, as the accept-ch preferences are passed down to 1p subresources. We're also collecting use-cases and feedback about it, so if you have concrete examples of the issue we'd love to know!

 b) I'm a little unclear on mobileness, what values should be given for a phone, tablet, desktop?

 

If you're asking about the content of the header itself, it's based on the Structured Headers proposal here. As for how to choose which value, we're going on what Chrome currently uses, which is to say that desktops send false ("?0") and Android-based devices send true ("?1"), unless "Request Desktop" is enabled, in which case it sends false. This is the current implicit definition of "mobileness", used for whether or not "Mobile" is put in the UA header string. 

c) Depending on b), it would be nice to at least have resolution buckets so we can return the most appropriate experience. Right now the experience for universal applications having to leverage UA sniffing (server) and resolution (client) is really bad.

FYI, there is also the "viewport-width" client hint, which will tell you the CSS px width of the viewport. If you want to go even crazier, you can request the "model" client hint, get the device model, and look up the width, height, and any other information based on that. That's not as recommended though unless you need very fine-grained information, as it's always better to get the least amount of entropy needed.

On Tuesday, January 14, 2020 at 5:22:00 AM UTC-6, Yoav Weiss wrote:

Contact emails

yoav...@chromium.org, aaro...@chromium.org

Summary

We want to freeze and unify (but not remove) the User Agent string in HTTP requests as well as in `navigator.userAgent`

Motivation

The User-Agent string is an abundant source of passive fingerprinting information about our users. It contains many details about the user’s browser and device as well as many lies ("Mozilla/5.0", anyone?) that were or are needed for compatibility purposes, as servers grew reliant on bad User Agent sniffing.

On top of those privacy issues, User-Agent sniffing is an abundant source of compatibility issues, in particular for minority browsers, resulting in browsers lying about themselves (generally or to specific sites), and sites (including Google properties) being broken in some browsers for no good reason.

The above abuse makes it desirable to freeze the UA string and replace it with a better mechanism. There have been past attempts at UA string freezing from the Safari team, but without an alternative way to perform UA based content-negotiation, they had to be partially reverted.

 

The User Agent Clicsscssent Hints (UA-CH) feature provides an alternate source for the information the User-Agent string provides, both in its request header form as well as its JS API one. 

[Domenic Denicola]

From: blin...@chromium.org <blin...@chromium.org> On Behalf Of Yoav Weiss

> The assumption is that polyfill avoidance (and most other use-cases for UA string content negotiation) can get by with `Sec-CH-UA` values of brand and major version that will continue to be sent to servers by default.
> Is that not the case? If not, can you elaborate on the need to polyfill based on minor versions?

Safari generally adds significant features in minor versions, so I'm not sure it would suffice for their use case.

[con...@superhuman.com]

Thanks for sharing this. Maybe I missed this in reading through the spec, but is the same information as sent via the Sec-CH- headers going to be available to JavaScript running on the page?

I ask because at Superhuman we parse the userAgent string client side for several things:

1. If the operating system is Linux or Windows we increase the contrast of the CSS to better fit with user expectations.
2. Depending on the device (e.g. iPhone vs Android) we display a different error message to users, and to direct them to the appropriate App Store.
3. We use the minor version number of the browser to block known ‘bad’ versions of Chrome Canary and explain the situation (in the rare cases it contains bugs that break Superhuman fundamentally)

Currently we load the HTML and JavaScript from Google Cloud Storage, and so this logic could not live on the server for us.

[Rowan Merewood]

Hey Conrad,

I think it's a little clearer in the explainer: https://github.com/WICG/ua-client-hints#a-proposal proposes exposing a NavigatorUAData interface via navigator.getUserAgent()

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5e5bd3db-0662-4de7-8bbb-004091b780f9%40chromium.org.

[Yoav Weiss]

Indeed. I wouldn't consider Safari's mid-year releases a "minor version" though. We need to clarify that in the spec. 

[fer...@gmail.com]

> The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay.

This drawback seems significant and ought to block the release of this feature until it is solved.

Moving fast and breaking things is not the way of the web.

[Conrad Irwin]

Thanks Rowan!

Out of interest, is there a public timeline for client hints shipping without a flag? I would have thought that this would block on that, but it sounds like they're happening independently.

(Beyond https://bugs.chromium.org/p/chromium/issues/detail?id=928669 and https://www.chromestatus.com/feature/5995832180473856, is there anywhere else I can subscribe to updates?)

Conrad

P.S. Superhuman is hiring — referral bonus for Full Stack Engineers: $1,947.

On Tue, Jan 14, 2020 at 9:41 AM, Rowan Merewood <mere...@chromium.org> wrote:

Hey Conrad,

I think it's a little clearer in the explainer: https://github.com/WICG/ua-client-hints#a-proposal proposes exposing a NavigatorUAData interface via navigator.getUserAgent()

On Tue, 14 Jan 2020 at 17:08, conrad via blink-dev <blink-dev@chromium.org> wrote:

Thanks for sharing this. Maybe I missed this in reading through the spec, but is the same information as sent via the Sec-CH- headers going to be available to JavaScript running on the page?

I ask because at Superhuman we parse the userAgent string client side for several things:

1. If the operating system is Linux or Windows we increase the contrast of the CSS to better fit with user expectations.
2. Depending on the device (e.g. iPhone vs Android) we display a different error message to users, and to direct them to the appropriate App Store.
3. We use the minor version number of the browser to block known ‘bad’ versions of Chrome Canary and explain the situation (in the rare cases it contains bugs that break Superhuman fundamentally)

Currently we load the HTML and JavaScript from Google Cloud Storage, and so this logic could not live on the server for us.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[aaro...@chromium.org]

The I2S for the UA client hints is here: https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/A4wxFpvqUfA

TL;DR: M81

[Shimi Zhang]

How does this affect Android WebView? WebView allows app developers to set user agent string, will that still work after the freeze and deprecation? Cc'ing people who might be interested in this Intent to Deprecate and Freeze.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/5e5bd3db-0662-4de7-8bbb-004091b780f9%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8a0dbd52-2793-4398-ac15-bb9afe64769b%40chromium.org.

[Aaron Tagliaboschi]

Can you point where the API meets the //content code for this path? I'm fairly sure that this would overwrite the frozen string, but I can make sure for you. I'm just not as familiar with the webview part of the codebase.

You received this message because you are subscribed to a topic in the Google Groups "blink-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/blink-dev/-2JIRNMWJ7s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BjkKumWa7az2jgGLNbyXAkLyeAffLwdh0gOeN80boRZMc4yCA%40mail.gmail.com.

[Shimi Zhang]

On Tue, Jan 14, 2020 at 12:41 PM Aaron Tagliaboschi <aaro...@chromium.org> wrote:

Can you point where the API meets the //content code for this path? I'm fairly sure that this would overwrite the frozen string, but I can make sure for you. I'm just not as familiar with the webview part of the codebase.

The meet point is AwSettings::UpdateUserAgentLocked(), it calls WebContents::SetUserAgentOverride() and NavigationEntry::SetIsOverridingUserAgent()

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJwB4XarrBWCMpAPEkD3PCL29bwnzCQ2Ah%3DJK7JHwssRL5FQJQ%40mail.gmail.com.

[Changwan Ryu]

Just wanted to note that WPT results for client hint on Android WebView is not on par with Chrome on Android:

https://docs.google.com/spreadsheets/d/1_UkaHp7Y5Qus_ASBXlGMOLKsrgBPSrCkVhDkUBYBFr0/edit#gid=1642632070

which may or may not related to https://crbug.com/921655.

Do we need to fix these?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BjkKukk_NJbJZz7W-yrm3wUCkXfxEj6Mioe9Yxr94U_3%3DmFww%40mail.gmail.com.

[Aaron Tagliaboschi]

There's some flakiness in one of the tests. I've just put out a CL for it. Would it be possible to get a report for the tests after the merge so we can see where any failures are?

[Michaela Merz]

I am trying to understand that: The data about the browser is still available, but we have to send an additional request (for Sec-CH) to the user-agent? Or can the (Accept-CH) header be sent right away during the initial server response? How and why is this preventing finger printing?

m.

On Tuesday, January 14, 2020 at 5:22:00 AM UTC-6, Yoav Weiss wrote:

[thorsten...@sap.com]

A  second roundtrip to get more information is definitely a performance issue especially with high latency times. From our perspective the platform e.g. win or macos would be helpful as default information as certain things behaves differently on the platform. Just to make a very simple example tooltips/menu entries for hotkeys (like ctrl+s) are differently on a mac or win (ctrl vs. cmd key) and already part from server in the first roundtrip.

Depending whether the frozen UserAgent string and JS interface remains it is highly incompatible and would result especially for enterprise customer to upgrade the complete landscape. 3 months are maybe enough to implement but not to update a large onprem landscape with hundreds of mission critical servers. We currently have this issue with the dismissal of syncXHR.

Enough time between releasing the new interface and deprecation of the old behavior would be necessary to give enterprise customers enough time to upgrade their systems or like in the syncXHR a possibility to opt-in the user-agent via flag and group policy

[jj]

Summary

We want to freeze and unify (but not remove) the User Agent string in HTTP requests as well as in `navigator.userAgent`

Just to state the obvious: This includes changes to navigator.appVersion and navigator.platform, yes?

 

[Philip Jägenstedt]

That's right, Safari currently has two releases per year. Between
Safari 12.0 and 12.1 there are significant changes both in WPT results
and API surface:
https://wpt.fyi/results/?label=master&label=stable&product=safari-12.0&product=safari-12.1&aligned&diff&filter=ADC
https://web-confluence.appspot.com/#!/catalog?releases=%5B%22Safari_12.0_OSX_10.13.6%22,%22Safari_12.1_OSX_10.13.6%22%5D&q=%22count:1%22

[Philip Jägenstedt]

I'm glad to see an effort towards freezing the UA string and finding alternatives. UA sniffing is at the root of countless web compat issues and these disproportionately affect browsers the developer didn't test in or think about.

At the same time, this header has been evolving and mutating since Mosaic, and freezing it is bound to have effects all across the ecosystem and cause a lot of frustration. I'd like to see some of the risks explored more. Some concrete concerns:

This depends on Intent to Ship: Client Hints infrastructure and UA Client Hints, but even if that ships first I don't think we can be confident that this alternative will be sufficient. Would it not be more prudent to ship that first, and work with a number of real sites to see what kind of issues come up? That's not going to shake out some of the long tail issues, but would reveal any common problems. Seeing other vendors adopt UA Client Hints would also help a lot.

The timeline for this seems a fair bit too short given that it's a non-trivial change and can span multiple layers of software, and so need trickle through multiple planning and release cycles. What are the tradeoff of letting this bake 6 months vs. 1 year, say? (Chrome 99 around mid-2022 seems like an upper bound because Chrome 100 will very likely break some sniffing code, just like how Opera's UA string got stuck at 9.)

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.) How essential is it to have this message? It looks like Safari doesn't have a console warning.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEjpz2AZ4x5nVm8X5%2BHrjdzH1-7f5y7qf6CmGMLTFVU4CQ%40mail.gmail.com.

[Yoav Weiss]

On Wed, Jan 15, 2020 at 11:45 AM Philip Jägenstedt <foo...@chromium.org> wrote:

I'm glad to see an effort towards freezing the UA string and finding alternatives. UA sniffing is at the root of countless web compat issues and these disproportionately affect browsers the developer didn't test in or think about.

At the same time, this header has been evolving and mutating since Mosaic, and freezing it is bound to have effects all across the ecosystem and cause a lot of frustration. I'd like to see some of the risks explored more. Some concrete concerns:

This depends on Intent to Ship: Client Hints infrastructure and UA Client Hints, but even if that ships first I don't think we can be confident that this alternative will be sufficient. Would it not be more prudent to ship that first, and work with a number of real sites to see what kind of issues come up?

It certainly would be more prudent. The urgency comes from the fact that right now, we are broadcasting a lot of information about our users, in clear text, to all servers. We would like to stop doing that.

Our assumption was that 4 releases (between shipping UA CH and unifying the UA string) would be enough time for us to make sure the replacement is sufficient before pulling the plug on the legacy string.

That's not going to shake out some of the long tail issues, but would reveal any common problems. Seeing other vendors adopt UA Client Hints would also help a lot.

I agree that other vendors adopting would help. Unfortunately, Mozilla don't seem to be interested in implementing the CH part. I've asked Apple to re-review, so one can remain hopeful.

The timeline for this seems a fair bit too short given that it's a non-trivial change and can span multiple layers of software, and so need trickle through multiple planning and release cycles. What are the tradeoff of letting this bake 6 months vs. 1 year, say? (Chrome 99 around mid-2022 seems like an upper bound because Chrome 100 will very likely break some sniffing code, just like how Opera's UA string got stuck at 9.)

Letting this bake for a year would mean continuing to send user information, in the clear, to all servers, for 6 more months.

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.)

If they'd use `getUserAgent()` for that, that's a win, no? :D

 

How essential is it to have this message? It looks like Safari doesn't have a console warning.

We want to let developers know that the API they are using is about to become useless and move them to the new API. Are there other tools that fit that purpose other than console warnings?

[Torne (Richard Coles)]

Some additional issues with WebView here:

1) I don't think that client hints are fully implemented in WebView right now (I don't think it is wired up to store the requests for hints from servers anywhere), so we might need to deal with that before shipping?

2) WebView apps which override the user agent string might be doing so in order to *conceal* information that's present by default, which might mean that they would also want to control what's exposed via client hints - we might need to consider exposing APIs for that.

3) In general many WebView apps aren't prepared for potentially-breaking changes to web-facing behaviour like this one, and the developers of those apps might not be participating in the larger web ecosystem in a way that would give them any visibility into these kinds of upcoming changes. This is not to say that we shouldn't make this change for WebView, but it may be riskier.

4) WebView's useragent string format is prescribed by Android's Compatibility Definition Document (CDD) and Compatibility Test Suite (CTS). We can't ship changes to WebView that would violate the rules set out in CDD/CTS unless we first update them, which in almost all cases is only permitted for *future* versions of Android not yet released, and can't be changed retroactively unless it's to fix actual errors in the definitions/tests. Freezing most aspects of the UA at their current values without changing the format would not violate the current implementation of CDD/CTS, with one exception: the Android version is expected to be the same value that's exposed to Android apps in general (i.e. something like "Android 5.1" - not a specific build ID or anything, just the "marketing" version number of the OS).

5) There's been past discussion about removing the Android device model name from the UA for either/both of WebView and Chrome on Android, but last I checked this hasn't been resolved. Will the freezing proposal have any impact on this?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJwB4XaR92xCbfp3emxkfRHG7y2Ns%3DCLYJcvL1eXv3oCw7-_KA%40mail.gmail.com.

[jj]

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.)

If they'd use `getUserAgent()` for that, that's a win, no? :D

 A lot of UA sniffing code will be happy with the frozen String for years.

Think on things like that, not written by the "sophisticated developers" mentioned in the spec:

if(/Mobi/.test(navigator.userAgent) && !/^iPad/.test(navigator.platform) || window.operamini) {
  /* linkify phone numbers */
}
No need to change this code, no reason for a spammy warning.

[Aaron Tagliaboschi]

That is correct. Interestingly, "appVersion" is based on the UA string itself, so if it's overridden it will be based on that. "platform" is not, but will send the equivalent frozen platform.

[Mike Sherov]

Apologies if this is the wrong place to post feedback, please point me to somewhere more appropriate if it exists.

I have a few points of feedback:

1. It seems here as if there are still unanswered questions about how to achieve parity with the features provided by the UA like how to get browser version on the first pageload, which seems like a very important use case.
2. The use case of preventing fine grained fingerprinting via CH is a good goal, but can't that be addressed by freezing the UA except for vendor, OS, and version, similar to Safari?
3. The public consensus section seems to indicate that FF and Safari have possibly different approaches in play or in mind, that would be compatible with partial UA freeze but not with CH.
4. The fact that freezing the UA is BC is good, but there's tons of existing material and libraries for parsing UA. Switching browser detection mechanism is a significant piece of churn for the Platform, and thus should come with a big benefit that can't be achieved otherwise.

Taking these points all together, it seems the main benefit of switching to CH over partial UA freeze is "drops legacy bits of text from the string" and "provides better ergonomics for sniffing". Neither of these IMO seem like a high enough bar to introduce a second system here and invalidate all of the existing libraries and documentation on the web. Secondly, it would seem you should solve the "first pageload" problem first.

Thanks for considering these points, and I'm happy to hear counterpoints and be convinced this feature is worth the churn.

[john...@google.com]

From a light reading, this seem detrimental to the "Chromium forks". There are many rebrandings of "Chromium" (Edge and Opera being some of the more prominent examples) that we simply ignore and use the Chromium versioning. It seems, as I've read it, to force browsers forks to masquerade as Chrome if they can't be recognized as part of the Chromium family. Did I miss something that would allow this grouping of "families"? Generally, server side rendering requires a reasonable user-agents identification.

I'm also concerned that "freezing" the UA prevents folks from getting the feedback that you need to validate that you have solved the right problems (as nothing is broken "today").

[she...@sher.pl]

How do you plan to support feature detection for Server Side Rendering? Client Hints are available only on second request

[abe...@fastly.com]

FYI polyfill.io currently serves 38 billion requests per month, which depend on accurately identifying browsers (including minot version) via the UA string.

[josh.d...@realtruck.com]

a) If the server doesn't get the info on first hit isn't there a performance penalty in that it has to respond with what it wants, then wait for the response just to give the optimal experience?

b) I'm a little unclear on mobileness, what values should be given for a phone, tablet, desktop?

c) Depending on b), it would be nice to at least have resolution buckets so we can return the most appropriate experience. Right now the experience for universal applications having to leverage UA sniffing (server) and resolution (client) is really bad.

On Tuesday, January 14, 2020 at 5:22:00 AM UTC-6, Yoav Weiss wrote:

Contact emails

yoav...@chromium.org, aaro...@chromium.org

Summary

We want to freeze and unify (but not remove) the User Agent string in HTTP requests as well as in `navigator.userAgent`

[josh.d...@realtruck.com]

On Tuesday, January 14, 2020 at 11:02:34 AM UTC-6, Aaron Tagliaboschi wrote:

FYI, there is also the "viewport-width" client hint, which will tell you the CSS px width of the viewport. If you want to go even crazier, you can request the "model" client hint, get the device model, and look up the width, height, and any other information based on that. That's not as recommended though unless you need very fine-grained information, as it's always better to get the least amount of entropy needed.

I'm confused.  I don't see anything about viewport in the User Agent Client Hints proposal https://wicg.github.io/ua-client-hints/ 

[Aaron Tagliaboschi]

Taking a look around, this should definitely overwrite the frozen string.

Aaron Tagliaboschi | Software Engineer | aaro...@google.com

[she...@sher.pl]

How do you plan to support feature detection for Server Side Rendering? Client Hints as currently designed won't work because they are available only on second request

[Aaron Tagliaboschi]

Aaron Tagliaboschi | Software Engineer | aaro...@google.com

On Tue, Jan 14, 2020 at 4:48 PM Michaela Merz <misc...@googlemail.com> wrote:

I am trying to understand that: The data about the browser is still available, but we have to send an additional request (for Sec-CH) to the user-agent? Or can the (Accept-CH) header be sent right away during the initial server response?

This can be requested in the initial response 

How and why is this preventing finger printing?

 

First off, it's not being sent unless it's asked for, which is nice. But more importantly it gives the opportunity for browsers and users to choose not to send the information for whatever reason (e.g. user-settings, lists, Privacy Budget), as well as being aware of what data is being requested.

See also: Intent to Ship: Client Hints infrastructure and UA Client Hints

[Josh Deltener]

Hmm, nevermind I see what you mean

https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/client-hints

The problem i see with this is, it doesn't really allow universal apps to return the best initial payload from the server for new users.

---

From: josh.d...@realtruck.com <josh.d...@realtruck.com>
Sent: Tuesday, January 14, 2020 11:21 AM
To: blink-dev <blin...@chromium.org>
Cc: Josh Deltener <Josh.D...@realtruck.com>; yoav...@chromium.org <yoav...@chromium.org>
Subject: Re: Intent to Deprecate and Freeze: The User-Agent string

 

[James Rosewell - 51Degrees]

Hi Yoav,

Whilst the change may be sensible from an engineering perspective - the User-Agent string is not efficient - it's going to be a breaking change for many industries and services that don't operate to browser provider "dog year" timescales.

Any technology being "retired" or regulatory change would typically be acompanied with a consultaiton period and two years notice. Consider GDPR or mobile networks.

Here's some examples:

1. Programmatic advertising. It's many players - including Google - have spent years developing message formats that have the User-Agent embedded in them. Upgrading them to client hints will require a major version change. The last major version change to AdCom / OpenRTB was released in November 2019 after many years consultation.

2. Analytics solutions will need to rush through changes to support client hints. Their users will need to migrate their deployments otherwise false assumptions will be formed wasting effort and causing confusion.

3. Performance improvements that require instant knowledge of the user agent to minimise data and improve render time for certain devices would be compromised due to the handshake. Consider people living in India where average device profiles are very different to western europe or north america.

4. Any implementation that removes the informaiton provided from the majority, whilst enabling Google due to it's size, influence or breadth of services (play store, android, search), to become the defacto single source of information about browser, operating system and device model usage globally risks being anti competitive. The implementaiton details don't appear to be clear enough to form a conclusion on this.

If the core problem is "we are broadcasting a lot of information about our users, in clear text, to all servers" then a staged approach might be to strengthen the warning around non SSL secure web sites and thrid parties to increase users control and awareness. Such a change could be accompanied with user experience monitoring (I opt in) and the severity of the problem better understood. Adding an SSL certificate is a change many web site operators have already made or are planning to make.

I'm interested in the subject as my business (51Degrees) provides device detection services utilising User-Agent for web traffic. TAC and app keys are used for non web. We've come a long way from "sniffing" using machine learning and other techniques to support analytics, optimisation and problem diagnosis. Switching to client hints helps us from an engineering perspective. Rolling out the change to client hints - in parallel with User-Agent - to gather the extra evidence fields is a substantial change.

Regards,

James

[ad...@gamerperfection.com]

Yoav,

Would the freezing of the UA and introduction of Client Hints assist in anyway of combating the fraudulent use of stolen fingerprints? I am assuming that the server requests to the browser for more detailed information which the "illicit" browser would provide would therefore not go towards helping prevent such fraudulent use?

If this is the case, are there any plans to help prevent this type of activity?

On Tuesday, 14 January 2020 11:22:00 UTC, Yoav Weiss wrote:

Contact emails

yoav...@chromium.org, aaro...@chromium.org

[Philip Jägenstedt]

On Wed, Jan 15, 2020 at 9:42 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Wed, Jan 15, 2020 at 11:45 AM Philip Jägenstedt <foo...@chromium.org> wrote:

I'm glad to see an effort towards freezing the UA string and finding alternatives. UA sniffing is at the root of countless web compat issues and these disproportionately affect browsers the developer didn't test in or think about.

At the same time, this header has been evolving and mutating since Mosaic, and freezing it is bound to have effects all across the ecosystem and cause a lot of frustration. I'd like to see some of the risks explored more. Some concrete concerns:

This depends on Intent to Ship: Client Hints infrastructure and UA Client Hints, but even if that ships first I don't think we can be confident that this alternative will be sufficient. Would it not be more prudent to ship that first, and work with a number of real sites to see what kind of issues come up?

It certainly would be more prudent. The urgency comes from the fact that right now, we are broadcasting a lot of information about our users, in clear text, to all servers. We would like to stop doing that.

Our assumption was that 4 releases (between shipping UA CH and unifying the UA string) would be enough time for us to make sure the replacement is sufficient before pulling the plug on the legacy string.

It could be enough time to show that migrating is possible in most scenarios before the UA string freezes, but it would also be unsurprising if trying to migrate a real-world complex case will reveal a number of issues that need to be resolved. Before we have confidence in CH as a replacement, I don't think we should be deciding on a timeline or communicating dates for UA freezing to web developers.

That's not going to shake out some of the long tail issues, but would reveal any common problems. Seeing other vendors adopt UA Client Hints would also help a lot.

I agree that other vendors adopting would help. Unfortunately, Mozilla don't seem to be interested in implementing the CH part. I've asked Apple to re-review, so one can remain hopeful.

Paging +Michael Taylor to clarify the linked issue. Would Mozilla consider freezing the UA string and only provide the same information via JS APIs? That seems pretty risky to me...

The timeline for this seems a fair bit too short given that it's a non-trivial change and can span multiple layers of software, and so need trickle through multiple planning and release cycles. What are the tradeoff of letting this bake 6 months vs. 1 year, say? (Chrome 99 around mid-2022 seems like an upper bound because Chrome 100 will very likely break some sniffing code, just like how Opera's UA string got stuck at 9.)

Letting this bake for a year would mean continuing to send user information, in the clear, to all servers, for 6 more months.

Yes, it's 6 more months, but what is the tradeoff? How bad for users is the UA string vs. how many sad developers and broken sites? I get there's no exact answer, but the impact on the ecosystem looks like the more concrete risk. It should also give us pause if other vendors aren't planning to freeze their UA strings, it calls the urgency into question.

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.)

If they'd use `getUserAgent()` for that, that's a win, no? :D

 

How essential is it to have this message? It looks like Safari doesn't have a console warning.

We want to let developers know that the API they are using is about to become useless and move them to the new API. Are there other tools that fit that purpose other than console warnings?

To spell out the concern, it is about warning fatigue. However, I have to agree that if we commit to a timeline, then we'll have to broadcast this very widely. A lot of the usage won't be practical to remove, however, so it would have to be for a limited time.

[Rik Cabanier]

Can you give some more specific examples what this freezing and unifying of the UA string will look like?

You imply that ALL desktop and mobile versions of Chrome will return the exact same desktop or mobile string regardless of platform (ie windows vs mac). Is that the case?

[Yoav Weiss]

On Wed, Jan 15, 2020 at 4:51 PM Torne (Richard Coles) <to...@chromium.org> wrote:

Some additional issues with WebView here:

1) I don't think that client hints are fully implemented in WebView right now (I don't think it is wired up to store the requests for hints from servers anywhere), so we might need to deal with that before shipping?

That's a good point, and indeed something we need to make sure happens before freezing, at least in WebViews.

 

2) WebView apps which override the user agent string might be doing so in order to *conceal* information that's present by default, which might mean that they would also want to control what's exposed via client hints - we might need to consider exposing APIs for that.

Interesting. What would exposing such an API entail?

 

3) In general many WebView apps aren't prepared for potentially-breaking changes to web-facing behaviour like this one, and the developers of those apps might not be participating in the larger web ecosystem in a way that would give them any visibility into these kinds of upcoming changes. This is not to say that we shouldn't make this change for WebView, but it may be riskier.

What would be a good way to increase awareness to this?

4) WebView's useragent string format is prescribed by Android's Compatibility Definition Document (CDD) and Compatibility Test Suite (CTS). We can't ship changes to WebView that would violate the rules set out in CDD/CTS unless we first update them, which in almost all cases is only permitted for *future* versions of Android not yet released, and can't be changed retroactively unless it's to fix actual errors in the definitions/tests. Freezing most aspects of the UA at their current values without changing the format would not violate the current implementation of CDD/CTS, with one exception: the Android version is expected to be the same value that's exposed to Android apps in general (i.e. something like "Android 5.1" - not a specific build ID or anything, just the "marketing" version number of the OS).

OK, so this may be somewhere where the WebView's UA string and Chrome's would have to diverge...

 

5) There's been past discussion about removing the Android device model name from the UA for either/both of WebView and Chrome on Android, but last I checked this hasn't been resolved. Will the freezing proposal have any impact on this?

This proposal intends to unify Android devices under a single common UA string, so under a single "model". Maybe we can get away (compatibility-wise) with removing the model entirely.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEV-rjf-bMK9sUqYgZ8PQ1V2GH8EeFChMEJ7w_JnUnoJqgfFng%40mail.gmail.com.

[Yoav Weiss]

On Wed, Jan 15, 2020 at 8:34 PM johnlenz via blink-dev <blin...@chromium.org> wrote:

From a light reading, this seem detrimental to the "Chromium forks". There are many rebrandings of "Chromium" (Edge and Opera being some of the more prominent examples) that we simply ignore and use the Chromium versioning.  It seems, as I've read it, to force browsers forks to masquerade as Chrome if they can't be recognized as part of the Chromium family.  Did I miss something that would allow this grouping of "families"?   Generally, server side rendering requires a reasonable user-agents identification. 

See https://github.com/WICG/ua-client-hints#should-the-ua-string-be-a-set and https://github.com/WICG/ua-client-hints/issues/22 

I'm also concerned that "freezing" the UA prevents folks from getting the feedback that you need to validate that you have solved the right problems (as nothing is broken "today").

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/7b2fd069-ab61-4257-bc8c-0b5d73a6bb1e%40chromium.org.

[Yoav Weiss]

While I'd have preferred such uses to move to getUserAgent() (cheaper and more accurate), I see your point.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/62082d91-ce30-4455-806a-0b68eee531ab%40chromium.org.

[Yoav Weiss]

On Tue, Jan 14, 2020 at 10:51 PM <thorsten...@sap.com> wrote:

A  second roundtrip to get more information is definitely a performance issue especially with high latency times. From our perspective the platform e.g. win or macos would be helpful as default information as certain things behaves differently on the platform. Just to make a very simple example tooltips/menu entries for hotkeys (like ctrl+s) are differently on a mac or win (ctrl vs. cmd key) and already part from server in the first roundtrip.

Sending different code to each platform just to handle different keys sounds like an overkill. Do you have a real-life example where OS related changes can't be reasonably done on the client-side?

 

Depending whether the frozen UserAgent string and JS interface remains it is highly incompatible and would result especially for enterprise customer to upgrade the complete landscape. 3 months are maybe enough to implement but not to update a large onprem landscape with hundreds of mission critical servers. We currently have this issue with the dismissal of syncXHR.

Enough time between releasing the new interface and deprecation of the old behavior would be necessary to give enterprise customers enough time to upgrade their systems or like in the syncXHR a possibility to opt-in the user-agent via flag and group policy

Adding an enterprise policy flag sounds reasonable. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/98bb9c26-91c5-47b6-8165-dea0b2b4c7aa%40chromium.org.

[Yoav Weiss]

On Tue, Jan 14, 2020 at 7:26 PM <fer...@gmail.com> wrote:

> The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay.

This drawback seems significant and ought to block the release of this feature until it is solved.

The thought is that the parts that are sent by default (UA brand and meaningful version, as well as most-probably the mobileness bit) would be sufficient for most cases of UA based content negotiation to not require any delays.

Do you have common use-cases where this information will not be sufficient?

 

Moving fast and breaking things is not the way of the web.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d7deba3b-2075-4759-81b5-e0c41b944539%40chromium.org.

[Yoav Weiss]

On Wed, Jan 15, 2020 at 8:34 PM Mike Sherov <mike....@gmail.com> wrote:

Apologies if this is the wrong place to post feedback, please point me to somewhere more appropriate if it exists.

I have a few points of feedback:

1. It seems here as if there are still unanswered questions about how to achieve parity with the features provided by the UA like how to get browser version on the first pageload, which seems like a very important use case.

For the meaningful part of the browser version, it will be sent by default without requiring an opt-in.

For the full version, you are correct at the moment. If you referred to the latter, what would be the use-case? Why do you need that information for HTML content adaptation?

 

2. The use case of preventing fine grained fingerprinting via CH is a good goal, but can't that be addressed by freezing the UA except for vendor, OS, and version, similar to Safari?

The OS adds significant entropy. For vendor and version, you've successfully convinced me that we can leave them unfrozen for now, and potentially tackle them (in case we'd see a compatibility need) in the future.

3. The public consensus section seems to indicate that FF and Safari have possibly different approaches in play or in mind, that would be compatible with partial UA freeze but not with CH.
4. The fact that freezing the UA is BC is good, but there's tons of existing material and libraries for parsing UA. Switching browser detection mechanism is a significant piece of churn for the Platform, and thus should come with a big benefit that can't be achieved otherwise.

Taking these points all together, it seems the main benefit of switching to CH over partial UA freeze is "drops legacy bits of text from the string" and "provides better ergonomics for sniffing". Neither of these IMO seem like a high enough bar to introduce a second system here and invalidate all of the existing libraries and documentation on the web. Secondly, it would seem you should solve the "first pageload" problem first.

The main benefit is to provide developers that require more bits of entropy with an alternative mechanism that enables them to request it.

 

Thanks for considering these points, and I'm happy to hear counterpoints and be convinced this feature is worth the churn.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/927e2f0a-bbc1-4a2c-9cbe-ea05f59afb78%40chromium.org.

[Yoav Weiss]

On Wed, Jan 15, 2020 at 8:34 PM abetts via blink-dev <blin...@chromium.org> wrote:

FYI polyfill.io currently serves 38 billion requests per month, which depend on accurately identifying browsers (including minot version) via the UA string.

Note that the polyfill.io use case is well covered by the fact that UA brand and meaningful version are sent by default, without requiring an opt-in.

Regarding minor version, I believe there's some confusion around the term: https://github.com/WICG/ua-client-hints/issues/23

We will modify the spec to clarify that Safari's mid-year releases (which include new features) are not considered "minor".

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d5087dcc-aa14-4907-85c2-085a5ca3b5b5%40chromium.org.

[Yoav Weiss]

On Wed, Jan 15, 2020 at 8:35 PM <ad...@gamerperfection.com> wrote:

Yoav,

Would the freezing of the UA and introduction of Client Hints assist in anyway of combating the fraudulent use of stolen fingerprints? I am assuming that the server requests to the browser for more detailed information which the "illicit" browser would provide would therefore not go towards helping prevent such fraudulent use?

If this is the case, are there any plans to help prevent this type of activity?

Once we turn the information from being passively sent out to one that's actively asked for by the server, the browser could keep track of who is asking for it. Then a mechanism like the Privacy Budget could be used to restrict access to that information in cases where the browser suspects that the access to the information is for nefarious purposes.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/4733a289-7417-4c96-8758-ef1578ca2142%40chromium.org.

[Philip Jägenstedt]

On Fri, Jan 17, 2020 at 11:56 AM Yoav Weiss <yo...@yoav.ws> wrote:

On Tue, Jan 14, 2020 at 7:26 PM <fer...@gmail.com> wrote:

> The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay.

This drawback seems significant and ought to block the release of this feature until it is solved.

The thought is that the parts that are sent by default (UA brand and meaningful version, as well as most-probably the mobileness bit) would be sufficient for most cases of UA based content negotiation to not require any delays.

Do you have common use-cases where this information will not be sufficient?

Learning about such use cases would be valuable, but I don't think we can assume the change to be safe in the absence of such examples. And at the scale we're talking about, usage doesn't need to be common to pose a serious compat problem.

Given that use counters will fail us here, some different form of compat analysis is needed. https://gist.github.com/foolip/38e3ea6f060bad4638c2aea6fac63e80 is an example of what this might look like, there varying the Accept-Language header and looking at how the result varies. Rough sketch of what it might look like for the UA header, in line with Blink principles of web compatibility:

• Crawl the same sites with multiple forms of the UA header
• Identify the sites where some of the responses vary (could also compare code paths taken and rendering)
• Confirm the differences by re-crawling to reduce noise
• Look at a sample of the candidate sites, assessing the severity to users and the difficulty to fix for users

This is time consuming, but it's what it will take to understand the risk.

[Yoav Weiss]

On Thu, Jan 16, 2020 at 4:22 PM Philip Jägenstedt <foo...@chromium.org> wrote:

On Wed, Jan 15, 2020 at 9:42 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Wed, Jan 15, 2020 at 11:45 AM Philip Jägenstedt <foo...@chromium.org> wrote:

I'm glad to see an effort towards freezing the UA string and finding alternatives. UA sniffing is at the root of countless web compat issues and these disproportionately affect browsers the developer didn't test in or think about.

At the same time, this header has been evolving and mutating since Mosaic, and freezing it is bound to have effects all across the ecosystem and cause a lot of frustration. I'd like to see some of the risks explored more. Some concrete concerns:

This depends on Intent to Ship: Client Hints infrastructure and UA Client Hints, but even if that ships first I don't think we can be confident that this alternative will be sufficient. Would it not be more prudent to ship that first, and work with a number of real sites to see what kind of issues come up?

It certainly would be more prudent. The urgency comes from the fact that right now, we are broadcasting a lot of information about our users, in clear text, to all servers. We would like to stop doing that.

Our assumption was that 4 releases (between shipping UA CH and unifying the UA string) would be enough time for us to make sure the replacement is sufficient before pulling the plug on the legacy string.

It could be enough time to show that migrating is possible in most scenarios before the UA string freezes, but it would also be unsurprising if trying to migrate a real-world complex case will reveal a number of issues that need to be resolved. Before we have confidence in CH as a replacement, I don't think we should be deciding on a timeline or communicating dates for UA freezing to web developers.

That's not going to shake out some of the long tail issues, but would reveal any common problems. Seeing other vendors adopt UA Client Hints would also help a lot.

I agree that other vendors adopting would help. Unfortunately, Mozilla don't seem to be interested in implementing the CH part. I've asked Apple to re-review, so one can remain hopeful.

Paging +Michael Taylor to clarify the linked issue. Would Mozilla consider freezing the UA string and only provide the same information via JS APIs? That seems pretty risky to me...

The timeline for this seems a fair bit too short given that it's a non-trivial change and can span multiple layers of software, and so need trickle through multiple planning and release cycles. What are the tradeoff of letting this bake 6 months vs. 1 year, say? (Chrome 99 around mid-2022 seems like an upper bound because Chrome 100 will very likely break some sniffing code, just like how Opera's UA string got stuck at 9.)

Letting this bake for a year would mean continuing to send user information, in the clear, to all servers, for 6 more months.

Yes, it's 6 more months, but what is the tradeoff? How bad for users is the UA string vs. how many sad developers and broken sites? I get there's no exact answer, but the impact on the ecosystem looks like the more concrete risk.

IMO, the main risk for breakage is from the OS and device unification. 

The former seems less risky for server side detection, but maybe carries some risk on the client side (e.g. the wrong keyboard shortcuts picked)

For the latter, my assumption was that folks that are tailoring their layouts to specific devices have to remain on their toes and update that constantly, as new devices arrive to market all the time. 

It should also give us pause if other vendors aren't planning to freeze their UA strings, it calls the urgency into question.

Safari has already partially frozen their UA. Let's wait to see what Mike Taylor says, but my reading of Mozilla's position is that they're considering the same.

 

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.)

If they'd use `getUserAgent()` for that, that's a win, no? :D

 

How essential is it to have this message? It looks like Safari doesn't have a console warning.

We want to let developers know that the API they are using is about to become useless and move them to the new API. Are there other tools that fit that purpose other than console warnings?

To spell out the concern, it is about warning fatigue. However, I have to agree that if we commit to a timeline, then we'll have to broadcast this very widely. A lot of the usage won't be practical to remove, however, so it would have to be for a limited time.

Limited time warnings sound reasonable.

[Yoav Weiss]

That's indeed the intention.

[Yoav Weiss]

On Fri, Jan 17, 2020 at 6:22 AM Philip Jägenstedt <foo...@chromium.org> wrote:

On Fri, Jan 17, 2020 at 11:56 AM Yoav Weiss <yo...@yoav.ws> wrote:

On Tue, Jan 14, 2020 at 7:26 PM <fer...@gmail.com> wrote:

> The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay.

This drawback seems significant and ought to block the release of this feature until it is solved.

The thought is that the parts that are sent by default (UA brand and meaningful version, as well as most-probably the mobileness bit) would be sufficient for most cases of UA based content negotiation to not require any delays.

Do you have common use-cases where this information will not be sufficient?

Learning about such use cases would be valuable, but I don't think we can assume the change to be safe in the absence of such examples. And at the scale we're talking about, usage doesn't need to be common to pose a serious compat problem.

As a side note, I believe the concerns raised were not about compat, but about missing/lacking capability in the alternative.

With that said... 

Given that use counters will fail us here, some different form of compat analysis is needed. https://gist.github.com/foolip/38e3ea6f060bad4638c2aea6fac63e80 is an example of what this might look like, there varying the Accept-Language header and looking at how the result varies.

That's great research! Seems relevant for https://github.com/WICG/lang-client-hint

 

Rough sketch of what it might look like for the UA header, in line with Blink principles of web compatibility:

• Crawl the same sites with multiple forms of the UA header
• Identify the sites where some of the responses vary (could also compare code paths taken and rendering)
• Confirm the differences by re-crawling to reduce noise
• Look at a sample of the candidate sites, assessing the severity to users and the difficulty to fix for users

This is time consuming, but it's what it will take to understand the risk.

That sounds reasonable to me. 

[Philip Jägenstedt]

On Fri, Jan 17, 2020 at 12:24 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Thu, Jan 16, 2020 at 4:22 PM Philip Jägenstedt <foo...@chromium.org> wrote:

On Wed, Jan 15, 2020 at 9:42 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Wed, Jan 15, 2020 at 11:45 AM Philip Jägenstedt <foo...@chromium.org> wrote:

I'm glad to see an effort towards freezing the UA string and finding alternatives. UA sniffing is at the root of countless web compat issues and these disproportionately affect browsers the developer didn't test in or think about.

At the same time, this header has been evolving and mutating since Mosaic, and freezing it is bound to have effects all across the ecosystem and cause a lot of frustration. I'd like to see some of the risks explored more. Some concrete concerns:

This depends on Intent to Ship: Client Hints infrastructure and UA Client Hints, but even if that ships first I don't think we can be confident that this alternative will be sufficient. Would it not be more prudent to ship that first, and work with a number of real sites to see what kind of issues come up?

It certainly would be more prudent. The urgency comes from the fact that right now, we are broadcasting a lot of information about our users, in clear text, to all servers. We would like to stop doing that.

Our assumption was that 4 releases (between shipping UA CH and unifying the UA string) would be enough time for us to make sure the replacement is sufficient before pulling the plug on the legacy string.

It could be enough time to show that migrating is possible in most scenarios before the UA string freezes, but it would also be unsurprising if trying to migrate a real-world complex case will reveal a number of issues that need to be resolved. Before we have confidence in CH as a replacement, I don't think we should be deciding on a timeline or communicating dates for UA freezing to web developers.

That's not going to shake out some of the long tail issues, but would reveal any common problems. Seeing other vendors adopt UA Client Hints would also help a lot.

I agree that other vendors adopting would help. Unfortunately, Mozilla don't seem to be interested in implementing the CH part. I've asked Apple to re-review, so one can remain hopeful.

Paging +Michael Taylor to clarify the linked issue. Would Mozilla consider freezing the UA string and only provide the same information via JS APIs? That seems pretty risky to me...

The timeline for this seems a fair bit too short given that it's a non-trivial change and can span multiple layers of software, and so need trickle through multiple planning and release cycles. What are the tradeoff of letting this bake 6 months vs. 1 year, say? (Chrome 99 around mid-2022 seems like an upper bound because Chrome 100 will very likely break some sniffing code, just like how Opera's UA string got stuck at 9.)

Letting this bake for a year would mean continuing to send user information, in the clear, to all servers, for 6 more months.

Yes, it's 6 more months, but what is the tradeoff? How bad for users is the UA string vs. how many sad developers and broken sites? I get there's no exact answer, but the impact on the ecosystem looks like the more concrete risk.

IMO, the main risk for breakage is from the OS and device unification. 

The former seems less risky for server side detection, but maybe carries some risk on the client side (e.g. the wrong keyboard shortcuts picked)

For the latter, my assumption was that folks that are tailoring their layouts to specific devices have to remain on their toes and update that constantly, as new devices arrive to market all the time.

FWIW, my hunches align, OS information ought to be used more than device information. Both should be possible to explore with the crawling suggested elsewhere in this thread.

[Yoav Weiss]

In related discussions I was convinced that freezing the major version part of the UA string does not hold much privacy benefits (as this is information we intend to expose by default), and can provide more time for developers to move from UA string based content adaptation (that relies on the version number) to UA-CH.

Therefore I'd like to modify our proposal to indicate that at the first phase, instead of freezing the browser version, we would omit the browser's minor version, similar to the way Safari are partially freezing their browser version.

That is not to say that we won't try to freeze the version entirely in the future, if we'd see compat benefits to doing so (e.g. servers getting it wrong, causing compat issues).

[thorsten...@sap.com]

On Friday, January 17, 2020 at 5:53:50 AM UTC+1, Yoav Weiss wrote:

On Tue, Jan 14, 2020 at 10:51 PM <thorste...@sap.com> wrote:

A  second roundtrip to get more information is definitely a performance issue especially with high latency times. From our perspective the platform e.g. win or macos would be helpful as default information as certain things behaves differently on the platform. Just to make a very simple example tooltips/menu entries for hotkeys (like ctrl+s) are differently on a mac or win (ctrl vs. cmd key) and already part from server in the first roundtrip.

Sending different code to each platform just to handle different keys sounds like an overkill. Do you have a real-life example where OS related changes can't be reasonably done on the client-side?

 

The keycode was just an example. The issues are server side rendering frameworks which are used to do the stuff on the server side without many JS coding. First of all they must use more Javascript to get the things done on client-side. Of course this is manageable. 

But they also provide information derived from the user agent which are normally the browser type, version number and OS (at least in our cases) to the application. It depends what the application is doing with it but it is complete out of control of the framework provider. 

Considering the maintenance  window of > 10 years for business systems it is very hard to find all this cases and fix them especially as the application are also developed by customers.  I’m not arguing completely against such changes but it should somehow compliant or enough time to change the onprem systems of enterprise customers. 

This is the challenge as enterprise customer with mission critical systems thinks in years to upgrade the complete landscape whereas the browsers vendors thinks in months. Something in between would be great. 

Getting the correct OS in the first roundtrip would be helpful to reduce the risk of failures. 

 

Depending whether the frozen UserAgent string and JS interface remains it is highly incompatible and would result especially for enterprise customer to upgrade the complete landscape. 3 months are maybe enough to implement but not to update a large onprem landscape with hundreds of mission critical servers. We currently have this issue with the dismissal of syncXHR.

Enough time between releasing the new interface and deprecation of the old behavior would be necessary to give enterprise customers enough time to upgrade their systems or like in the syncXHR a possibility to opt-in the user-agent via flag and group policy

Adding an enterprise policy flag sounds reasonable. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

[Torne (Richard Coles)]

On Fri, 17 Jan 2020 at 04:41, Yoav Weiss <yo...@yoav.ws> wrote:

On Wed, Jan 15, 2020 at 4:51 PM Torne (Richard Coles) <to...@chromium.org> wrote:

Some additional issues with WebView here:

1) I don't think that client hints are fully implemented in WebView right now (I don't think it is wired up to store the requests for hints from servers anywhere), so we might need to deal with that before shipping?

That's a good point, and indeed something we need to make sure happens before freezing, at least in WebViews.

 

2) WebView apps which override the user agent string might be doing so in order to *conceal* information that's present by default, which might mean that they would also want to control what's exposed via client hints - we might need to consider exposing APIs for that.

Interesting. What would exposing such an API entail?

Designing what it should look like and implementing the API in the WebView AndroidX support library, which enables us to ship new APIs without waiting for a new Android OS release: https://developer.android.com/reference/androidx/webkit/package-summary

3) In general many WebView apps aren't prepared for potentially-breaking changes to web-facing behaviour like this one, and the developers of those apps might not be participating in the larger web ecosystem in a way that would give them any visibility into these kinds of upcoming changes. This is not to say that we shouldn't make this change for WebView, but it may be riskier.

What would be a good way to increase awareness to this?

We don't have an especially good answer right now, but we're working on it. 

 

4) WebView's useragent string format is prescribed by Android's Compatibility Definition Document (CDD) and Compatibility Test Suite (CTS). We can't ship changes to WebView that would violate the rules set out in CDD/CTS unless we first update them, which in almost all cases is only permitted for *future* versions of Android not yet released, and can't be changed retroactively unless it's to fix actual errors in the definitions/tests. Freezing most aspects of the UA at their current values without changing the format would not violate the current implementation of CDD/CTS, with one exception: the Android version is expected to be the same value that's exposed to Android apps in general (i.e. something like "Android 5.1" - not a specific build ID or anything, just the "marketing" version number of the OS).

OK, so this may be somewhere where the WebView's UA string and Chrome's would have to diverge...

Yeah, the specific details here might need to evolve over time, to give us the opportunity to tweak the CDD definitions to bring it more in line with Chrome.

 

5) There's been past discussion about removing the Android device model name from the UA for either/both of WebView and Chrome on Android, but last I checked this hasn't been resolved. Will the freezing proposal have any impact on this?

This proposal intends to unify Android devices under a single common UA string, so under a single "model". Maybe we can get away (compatibility-wise) with removing the model entirely.

The current CTS/CDD allows the model to be absent but doesn't allow it to be "fake" - if present it has to match the model string available to native apps via android APIs. So, at least for WebView we would actually need to remove it rather than unify it. But, even though it's optional in the compatibility definition at the moment, in practise it's always sent unless the device is running a preproduction release of Android (to avoid potentially leaking unreleased device info), so apps/sites may not really handle it being absent gracefully. :/

I chatted with Aaron and he suggested we VC internally to discuss more of these details and how the WebView team can help.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEjYiVxS044Ns7D1BfTqLs4XPBo0R8prtS5bmdTkw5ZZmw%40mail.gmail.com.

[fri...@jeka.info]

Am Freitag, 17. Januar 2020 05:48:04 UTC+1 schrieb Yoav Weiss:

On Wed, Jan 15, 2020 at 4:55 PM jj <fri...@jeka.info> wrote:

Showing a deprecation message on 90% of page views is a lot. Sites will likely keep accessing navigator.userAgent if they've determined nothing will break, or they'd have to use another form of sniffing to avoid it. (I have found UA sniffing to avoid triggering deprecate messages in past httparchive digging.)

If they'd use `getUserAgent()` for that, that's a win, no? :D

 A lot of UA sniffing code will be happy with the frozen String for years.

Think on things like that, not written by the "sophisticated developers" mentioned in the spec:

if(/Mobi/.test(navigator.userAgent) && !/^iPad/.test(navigator.platform) || window.operamini) {
  /* linkify phone numbers */
}
No need to change this code, no reason for a spammy warning.

While I'd have preferred such uses to move to getUserAgent() (cheaper and more accurate), I see your point.

You can't move to getUserAgent() if your target don't know it.

If(document.documentMode || /Edge\//.test(navigator.userAgent) {
  /* double box-shadow blur values for IE an EdgeHTML to match other browsers */
}

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blin...@chromium.org.

[jj]

hmm, don't you need to freeze the version number in any case soonish? How to avoid a version 99->100 disaster otherwise?

[jwros...@googlemail.com]

Hi Yoav,

As stakeholders in the web and following discussions with colleagues, clients and industry professionals I’ve prepared a summary of questions and observations concerning your proposed changes to the established usage of the HTTP User-Agent header field. This usage has been established since the beginning of the web and the purpose of the field is noted in the documentation of HTTP from 1990s. 

While anybody must be concerned about possibly compromising the privacy of the users of the web we feel that the nature and extent of those compromises needs to be more clearly established. The effect of the proposed changes is to damage quite seriously the user experience enjoyed by users on a significant number of websites that require the User-Agent to be in its present form.

Further we are most concerned about the possible damaging effects on the accessibility of those sites and the availability of those sites in developing nations as a result of the proposed changes.

Here follows a number of more detailed comments and we’d be most grateful for your feedback on these comments.

1. Where is the evidence to support the primary assertion that the User-Agent is an “abundant source of passive fingerprinting information”?

 

Suggestion: A very large web site or network provider using session tracking - such as first party cookies -would be able to provide the analysis needed. Group a large cohort of web sessions by IP and User-Agent and count the number of sessions. Is the relation 1 to 1 or 1 to many? If 1 to many, how many? Does it vary by different formats of User-Agent? Is the analysis sufficient to validate the assertion?

2. In response to your second assertion User-Agent field values are poorly structured. It is our experience that a high degree of accuracy can be achieved from the current state of these strings.

We are not the only practitioners. Experts in the field of device identification and therefore User-Agents strings include:

 

51Degrees (this authors company “we”)

Device Atlas (owned by Afilias)

ScientiaMobile via the WURFL solution.

 

Between us support is provided to millions of web sites including blue chip businesses as varied as Google, Facebook, Oracle, eBay, Amazon, Disney, Tencent, IBM, Naspers, Verizon and countless others many of which operate outside North America and Europe.

 

We provide free and open source solutions. Access to first request device information is essential to the services they support. We have not been consulted about this change.

3. Has thought been given to the impact on non-technical web professionals such as marketing professionals who use analytics tools? As a result of the proposed changes their website may break. As a minimum their web site analytics will be incorrect. They will report incorrect information and form invalid conclusions. This will be frustrating to them and waste countless days head scratching.

4. Have network operators been consulted about the deployment of IPv6 and networks in general? The fingerprinting argument depends at least 50% on the IP address. Depending on deployment a client IP address could persist with a device forever; effectively forming a unique identifier. Frequently changing the client IP address will lead to inefficiencies for network operators. 

5. Has the change been validated with web site owners large and small who operate in many different markets and geographies, many of whom make extensive use of the User-Agent to deliver their services? How do developing countries differ to north America and Europe?

 

Consider news services from Naspers and Tencent targeting Asia and Africa. These services use the information obtained from the User-Agent in the very first request to optimise content ensuring feature phones, powerful smartphones and budget smartphones all receive optimised content cost effectively. Many consumers outside north America and Western Europe do not enjoy “all you can eat” data plans or high-end electronics. They need this feature to receive cost effective and optimised news.

 

Facebook Zero inspired many businesses to operate this “stripped down” approach. They tend not to be covered in the English language technology press or outside their home markets.

6. In our experience many websites are bloated with MBs of JavaScript, constrained by a single threaded execution engine, and perform poorly. Google invented Accelerated Mobile Pages (AMP) to offer readers a faster way of consuming publishers’ content. It is often seen deployed in Android where knowledge of the device informs the information presented. Companies that are not Google who host AMP will lack the necessary information to provide their service.

 

Many publishers including Reach, Mail Online and News International are still plagued by JavaScript performance issues over 5 years after Google shown a light on the performance problem.

 

From an engineering perspective optimising content on the web server at first request to reduce the amount of data transmitted and the computation the user agent needs to perform is very sensible. The client and web server work more harmoniously together rather than the server being a dumb file server. Making the information needed available in the first request is essential to enable this performance improvement. The User-Agent isn’t ideally formatted, but it works.

 

It works so well http://chromium.org uses the technique with Google Sites. The User-Agent is everywhere.

7. Does the process of agreeing this include a sufficiently broad cross section of engineers and web professionals? For example the User-Agent is used extensively within the HTTP protocol by CDNs and within edge networks both with and without SSL. Such engineers are key stakeholders in such a change. Should the IEFT be consulted? Is this not part of HTTP 3?

 

Current contributions appear to be skewed to be from people with a web front end / JavaScript centric view.

8. Is Safari a good comparator for this performance degrading change? Users are skewed towards higher socio-economic groups, high end chipsets and specific geographic markets. Chromium is the ubiquitous foundation of the web browser globally. It’s used with $10 MediaTek A53 chips to the most powerful Qualcomm SoCs.

9. Is Firefox a good comparator? In many cases Firefox is not bundled with Android by default. It is installed by a user who wishes to benefit from its features. As such a typical user is more privacy aware. As Chromium is the default for most Android devices it should support the widest set of requirements out of the box. Perhaps Google will offer a choice of web browser when Android is first setup? Microsoft did a similar thing in the past with Windows.

10. Is this a feature that should be up to the browser vendor to form a view on rather than sit within the Chromium project? Brave already take one approach, Microsoft another. Privacy conscious users can easily switch between browsers.

11. Public support seems to be in the form of other web browser engineers via a series of tweets and git issues posts operating within a “bubble” or “echo chamber”. This tweet is cited as support (https://twitter.com/_scottlow/status/1206831008261132289). Is this sufficient support for such a wide-reaching change impacting almost every digital citizen and business?

12. What is the view of “minority” browser vendors who benefit extensively via marketing and analytics by appending their browser details to the User-Agent string? What is the view of Android device vendors who wish to be able to gain insight into device usage?

13. Even if a large company has the resources to make the necessary changes within the timescales proposed what are the changes required to maintain the current fine grain of optimisation enjoyed by their users? The majority of small and medium sized businesses and other non-profit web site operators do not. They are all impacted.

 

They are not of the size and scale of the relatively small number of AdTech / tracking businesses who need to adapt to cookie changes. These companies were given 2 years notice.

 

This change impacts every website operator and owner. It is impractical to assume the vast majority can or want to make changes to their sites.

14. The AdTech business, of which Google is a major player, have spent a long time producing the standards used to communicate between each other in real time. Has an assessment been performed on the impact to OpenRTB and AdCom where User-Agent is the primary method of validating and unifying device information?

https://github.com/InteractiveAdvertisingBureau/AdCOM/blob/master/AdCOM%20v1.0%20FINAL.md#object_device

15. Where a change is needed there is an obvious solution? Consider the Y2K problem.  Like this proposed change it impacted every business.  The solution was clear; add two digits to the year and test everything.

 

In this case there are many use cases where a clear alternative does not exist. Consider the previously mentioned web site adapting on first request to the device with a granularity beyond just desktop or mobile. This change will remove this possibility and new designs will be required.

 

Or the web professionals wishing to understand why performance is better or worse by different users or web sessions. The IETF HTTP 1.1 specification states “This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations.”

 

https://tools.ietf.org/html/rfc2616

 

21 years later this is exactly how it’s being used. So widely the Chromium.org web site uses the User-Agent to adapt its content.

 

Have the IETF been consulted? Should this change be made by the IETF within the HTTP 3 specification?

16. Is feature detection really a good solution? If it were why does Chromium.org make use of the User-Agent? It’s not a very complex web site.

 

-       If 90% of web sites are making use of the User-Agent, as cited in the post – why change anything? It clearly works.

 

-       Given the dominance of Google services, and Google’s control over Chromium, the change will benefit Google economically by removing information from the wider web. Only Google via its dominance in Search, Android and the Play Store will understand device and browser usage. Will Google be making this data available in aggregate as open data? If not, then there exists an argument concerning abuse of dominate market position that Google would be left open to. Has this been considered?

There are a lot of implications to this proposal which do not as yet appear to have been thought through. We sincerely hope the governance model for such a change considers all stakeholders, particularly those who benefit from optimised content in developing countries and do not engage in these discussions.

Regards,

James Rosewell – For self and 51Degrees

[John Lenz]

I don't see that either of those links answer address my concerns that "is Chromium version X" is a significant attribute that needs to be served and that failure to do so will only lead to browsers completely lying about their identity and claiming to be "Chrome version X".    If this comes to pass this will pollute the Chrome brand as server logs will simply show "Chrome" and not "Brave", "Opera", "Edge", etc.

I have no reason to think that a new browser (Chromium fork or not) with no market share is going to somehow be able to update servers around the world to recognize it as being compatible as an existing browser.  They have every incentive to lie because failure to do so would also lead to a market failure and  any change in this space needs to recognize this reality.  Browsers don't masquerade as other browsers for fun but because they have to.

Server need to know who they are serving to for two reasons: (1) to know who they need to server (do I need to continue to support IE 11 and Chrome 23) and (2) how to maximize the user experience (do I serve ES6 polyfills and ES5 compatible code, can I assume feature X exists).    If you can't do this, you relegate the future of the web to the lowest common denominator of today. 

[ad...@gamerperfection.com]

On Friday, 17 January 2020 05:14:38 UTC, Yoav Weiss wrote:

On Wed, Jan 15, 2020 at 8:35 PM <ad...@gamerperfection.com> wrote:

Yoav,

Would the freezing of the UA and introduction of Client Hints assist in anyway of combating the fraudulent use of stolen fingerprints? I am assuming that the server requests to the browser for more detailed information which the "illicit" browser would provide would therefore not go towards helping prevent such fraudulent use?

If this is the case, are there any plans to help prevent this type of activity?

Once we turn the information from being passively sent out to one that's actively asked for by the server, the browser could keep track of who is asking for it. Then a mechanism like the Privacy Budget could be used to restrict access to that information in cases where the browser suspects that the access to the information is for nefarious purposes.

 

I see, but how about the other way round. What you mentioned is from the perspective of the website/server potentially being malicious. What if, in the case of a malicious actor that has stolen a fingerprint then uses that fingerprint when visiting a website that the victim is known to use (the actor may have stolen their credentials too). The genuine website may actively ask for the fuller information which the illicit browser provides. The genuine server believes this is the true customer and lets them through. Is there anything in the pipeline to combat this to improve websites anti-fraud detection?

[Brad Lassey]

I don't fully understand the threat you're describing (specifically "stealing a fingerprint"), but have a look at Trust Tokens which is a proposal to combat fraud and other abusive behavior through blinded tokens that can be granted in one context and redeemed in another without being able to link identity.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d45f46c1-8c49-4249-95e7-8d6d940d9a26%40chromium.org.

[Yoav Weiss]

On Sat, Jan 18, 2020 at 12:52 AM jwrosewell via blink-dev <blin...@chromium.org> wrote:

Hi Yoav,

As stakeholders in the web and following discussions with colleagues, clients and industry professionals I’ve prepared a summary of questions and observations concerning your proposed changes to the established usage of the HTTP User-Agent header field. This usage has been established since the beginning of the web and the purpose of the field is noted in the documentation of HTTP from 1990s. 

While anybody must be concerned about possibly compromising the privacy of the users of the web we feel that the nature and extent of those compromises needs to be more clearly established. The effect of the proposed changes is to damage quite seriously the user experience enjoyed by users on a significant number of websites that require the User-Agent to be in its present form.

Further we are most concerned about the possible damaging effects on the accessibility of those sites and the availability of those sites in developing nations as a result of the proposed changes.

Here follows a number of more detailed comments and we’d be most grateful for your feedback on these comments.

1. Where is the evidence to support the primary assertion that the User-Agent is an “abundant source of passive fingerprinting information”?

There’s plenty of research on the subject. For example, you could go to the EFF’s panopticlick, which personally, estimates my UA string entropy at 9.6 bits. That’s a lot of bits for us to be passively exposing. 

 

Suggestion: A very large web site or network provider using session tracking - such as first party cookies -would be able to provide the analysis needed. Group a large cohort of web sessions by IP and User-Agent and count the number of sessions. Is the relation 1 to 1 or 1 to many? If 1 to many, how many? Does it vary by different formats of User-Agent? Is the analysis sufficient to validate the assertion?

2. In response to your second assertion User-Agent field values are poorly structured. It is our experience that a high degree of accuracy can be achieved from the current state of these strings.

Maybe. But for the most part, we see use of UA strings in the wild that is brittle and results in bugs and compatibility issues. This is why Opera never had a UA string that claimed to be “Opera 10.0”. This is why MDN’s “Browser detection using the user agent” page is comprised of 60% asking the developer to consider alternatives, and the rest, guiding them through the minefield that is the parsing of the UA string. We also found a bunch of issues ([1] [2] [3]) on Mozilla's bug tracker which demonstrate how these issues are hitting users and browsers.

So, maybe individuals and companies that are entirely dedicated to this subject can achieve high accuracy (I’m doubtful), but in the common case, UA string parsing is complex and error-prone.

 

We are not the only practitioners. Experts in the field of device identification and therefore User-Agents strings include:

 

51Degrees (this authors company “we”)

Device Atlas (owned by Afilias)

ScientiaMobile via the WURFL solution.

 

Between us support is provided to millions of web sites including blue chip businesses as varied as Google, Facebook, Oracle, eBay, Amazon, Disney, Tencent, IBM, Naspers, Verizon and countless others many of which operate outside North America and Europe.

 

We provide free and open source solutions. Access to first request device information is essential to the services they support. We have not been consulted about this change.

3. Has thought been given to the impact on non-technical web professionals such as marketing professionals who use analytics tools? As a result of the proposed changes their website may break. As a minimum their web site analytics will be incorrect. They will report incorrect information and form invalid conclusions. This will be frustrating to them and waste countless days head scratching.

I'd expect analytics tools to adapt to this change. I'd also expect our decision to leave the meaningful version part of the UA string non-frozen for the time being to help provide them more time for that change.

For analytics beyond browser brand and version, those sites could opt-in to get more information (but shouldn't, as browsers clamping down on active fingerprinting will count those bits towards their e.g. Privacy Budget) 

4. Have network operators been consulted about the deployment of IPv6 and networks in general? The fingerprinting argument depends at least 50% on the IP address. Depending on deployment a client IP address could persist with a device forever; effectively forming a unique identifier. Frequently changing the client IP address will lead to inefficiencies for network operators. 

Other browsers have found ways to tackle IP privacy, so one can remain hopeful we would be able to avoid emitting those bits of passive entropy as well.  

5. Has the change been validated with web site owners large and small who operate in many different markets and geographies, many of whom make extensive use of the User-Agent to deliver their services? How do developing countries differ to north America and Europe?

 

Consider news services from Naspers and Tencent targeting Asia and Africa. These services use the information obtained from the User-Agent in the very first request to optimise content ensuring feature phones, powerful smartphones and budget smartphones all receive optimised content cost effectively. Many consumers outside north America and Western Europe do not enjoy “all you can eat” data plans or high-end electronics. They need this feature to receive cost effective and optimised news.

 

Facebook Zero inspired many businesses to operate this “stripped down” approach. They tend not to be covered in the English language technology press or outside their home markets.

There are multiple Client Hints that can help towards that goal: Viewport-Width, DPR as well as Device-Memory can help such sites serve appropriate resources to their customers based on their needs.
Such adaptation for everything that doesn't involve the navigation request of the very-first-view can be done without any extra performance cost. We're hopeful to provide a solution for the very-first-view as well soon.

 

6. In our experience many websites are bloated with MBs of JavaScript, constrained by a single threaded execution engine, and perform poorly. Google invented Accelerated Mobile Pages (AMP) to offer readers a faster way of consuming publishers’ content. It is often seen deployed in Android where knowledge of the device informs the information presented.

I'm not following the logical leap between bloated JS and the need to know the device model.  

Companies that are not Google who host AMP will lack the necessary information to provide their service.

The AMP cache and other Google properties will have to follow the same path as everyone else in case they need device model information to serve their content, and use UA-CH. FWIW, as far as I know, AMP does not require such info. 

 

Many publishers including Reach, Mail Online and News International are still plagued by JavaScript performance issues over 5 years after Google shown a light on the performance problem.

 

From an engineering perspective optimising content on the web server at first request to reduce the amount of data transmitted and the computation the user agent needs to perform is very sensible. The client and web server work more harmoniously together rather than the server being a dumb file server. Making the information needed available in the first request is essential to enable this performance improvement. The User-Agent isn’t ideally formatted, but it works.

 

It works so well http://chromium.org uses the technique with Google Sites. The User-Agent is everywhere.

7. Does the process of agreeing this include a sufficiently broad cross section of engineers and web professionals? For example the User-Agent is used extensively within the HTTP protocol by CDNs and within edge networks both with and without SSL.

CDNs will have to adapt as well. 

Such engineers are key stakeholders in such a change. Should the IEFT be consulted? Is this not part of HTTP 3?

The UA Client Hints started its life as an IETF draft. The Client Hints mechanism is defined as an HTTP extension draft in the HTTPWG, and has been thoroughly discussed.

 

 

Current contributions appear to be skewed to be from people with a web front end / JavaScript centric view.

I don’t think this claim is accurate. 

8. Is Safari a good comparator for this performance degrading change? Users are skewed towards higher socio-economic groups, high end chipsets and specific geographic markets. Chromium is the ubiquitous foundation of the web browser globally. It’s used with $10 MediaTek A53 chips to the most powerful Qualcomm SoCs.

9. Is Firefox a good comparator? In many cases Firefox is not bundled with Android by default. It is installed by a user who wishes to benefit from its features. As such a typical user is more privacy aware. As Chromium is the default for most Android devices it should support the widest set of requirements out of the box. Perhaps Google will offer a choice of web browser when Android is first setup? Microsoft did a similar thing in the past with Windows.

First, we don't believe that improved privacy should be limited to higher socio-economic groups or privacy aware users.

Second, I don’t think that the performance implications are significant. For most use-cases the browser’s name, meaningful version and mobileness are more than enough to perform the adaptations required. Devices indeed vary in their capabilities to process subresources (e.g. due to DPR differences, viewport dimensions or memory), which is why I worked on responsive images to enable responsive designs to conditionally load them. Client Hints also enable you to perform those adaptations on the server side, as I’m sure you know. 

10. Is this a feature that should be up to the browser vendor to form a view on rather than sit within the Chromium project? Brave already take one approach, Microsoft another. Privacy conscious users can easily switch between browsers.

Chromium embedders would be able to change that behavior, if they so wish, pretty much like any other part of the code. 

11. Public support seems to be in the form of other web browser engineers via a series of tweets and git issues posts operating within a “bubble” or “echo chamber”. This tweet is cited as support (https://twitter.com/_scottlow/status/1206831008261132289). Is this sufficient support for such a wide-reaching change impacting almost every digital citizen and business?

Yes. 

12. What is the view of “minority” browser vendors who benefit extensively via marketing and analytics by appending their browser details to the User-Agent string? What is the view of Android device vendors who wish to be able to gain insight into device usage?

Minority browsers tend to be hurt the most by UA string sniffing.  

13. Even if a large company has the resources to make the necessary changes within the timescales proposed what are the changes required to maintain the current fine grain of optimisation enjoyed by their users? The majority of small and medium sized businesses and other non-profit web site operators do not. They are all impacted.

 

They are not of the size and scale of the relatively small number of AdTech / tracking businesses who need to adapt to cookie changes. These companies were given 2 years notice.

 

This change impacts every website operator and owner. It is impractical to assume the vast majority can or want to make changes to their sites.

I don’t believe every website operator and owner is performing adaptation that is based on specific device models. Since we’re talking about here is freezing the UA-string, for all other types of adaptation (which is the vast majority of cases), no changes will be needed. 

14. The AdTech business, of which Google is a major player, have spent a long time producing the standards used to communicate between each other in real time. Has an assessment been performed on the impact to OpenRTB and AdCom where User-Agent is the primary method of validating and unifying device information?

https://github.com/InteractiveAdvertisingBureau/AdCOM/blob/master/AdCOM%20v1.0%20FINAL.md#object_device

 

Chrome has set out its principles for fingerprinting in our privacy model and various blog posts. The impacts to OpenRTP have been discussed in W3C meetings (RTB use case is an example artifact of such a discussion). It is probably more fruitful to discuss the implications of moving to a world that shards identity by first party in those forums than on any one given change that helps achieve it.

 

15. Where a change is needed there is an obvious solution? Consider the Y2K problem.  Like this proposed change it impacted every business.  The solution was clear; add two digits to the year and test everything.

 

In this case there are many use cases where a clear alternative does not exist. Consider the previously mentioned web site adapting on first request to the device with a granularity beyond just desktop or mobile. This change will remove this possibility and new designs will be required.

 

Or the web professionals wishing to understand why performance is better or worse by different users or web sessions. The IETF HTTP 1.1 specification states “This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations.”

 

https://tools.ietf.org/html/rfc2616

 

21 years later this is exactly how it’s being used. So widely the Chromium.org web site uses the User-Agent to adapt its content.

 

Have the IETF been consulted? Should this change be made by the IETF within the HTTP 3 specification?

Yes, as stated above, IETF WGs are well aware of this work. 

16. Is feature detection really a good solution? If it were why does Chromium.org make use of the User-Agent? It’s not a very complex web site.

 

Chromium.org indeed has a mobile and a desktop version, which seem to be switching based on the UA string. But exposing the mobileness bit by default would mean they can easily switch to use that mechanism instead (probably somewhat simplifying their backend code in the process). 

-       If 90% of web sites are making use of the User-Agent, as cited in the post – why change anything? It clearly works.

Two points here. First, it does work for a use case (fingerprinting) that we consider to be abusive of user choice and control, which necessitates intervention in and of itself. Second, it performs poorly for use cases we consider potentially non-harmful (and results in harm to users) as evidenced by the interventions that browsers have had to deploy to work around them.  

 

-       Given the dominance of Google services, and Google’s control over Chromium, the change will benefit Google economically by removing information from the wider web. Only Google via its dominance in Search, Android and the Play Store will understand device and browser usage. Will Google be making this data available in aggregate as open data? If not, then there exists an argument concerning abuse of dominate market position that Google would be left open to. Has this been considered?

While we disagree with the premise of your question, it is worth pointing out that device information would still be available to the wider web, only behind an opt-in. Properties that would be interested in collecting it would still be able to, assuming that they do not collect other bits of entropy that e.g. exceed their privacy budget. 

There are a lot of implications to this proposal which do not as yet appear to have been thought through. We sincerely hope the governance model for such a change considers all stakeholders, particularly those who benefit from optimised content in developing countries and do not engage in these discussions.

Regards,

James Rosewell – For self and 51Degrees

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/49485ca1-646e-4161-a6db-a2fd5c6396de%40chromium.org.

[Matthew Finkel]

On Friday, January 17, 2020 at 4:53:50 AM UTC, Yoav Weiss wrote:

On Tue, Jan 14, 2020 at 10:51 PM <thorste...@sap.com> wrote:

A  second roundtrip to get more information is definitely a performance issue especially with high latency times. From our perspective the platform e.g. win or macos would be helpful as default information as certain things behaves differently on the platform. Just to make a very simple example tooltips/menu entries for hotkeys (like ctrl+s) are differently on a mac or win (ctrl vs. cmd key) and already part from server in the first roundtrip.

Sending different code to each platform just to handle different keys sounds like an overkill. Do you have a real-life example where OS related changes can't be reasonably done on the client-side?

 

As a data point, a couple years ago Tor Browser developers found github.com and
Google Docs relied on the Javascript API for determining the client's OS, and
then fetched OS-specific client-side code for the hotkeys (among other
functionality, most likely). There are likely other sites that do this, as
well. The argument that the real OS must be advertised up-front in the

User-Agent header is weak.

Previously Tor Browser spoofed the client's OS in both the HTTP header and the
value returned in javascript. Tor Browser changed its behavior as a result of
the breakage caused by sites fetching OS-specific scripts relying on the UAS
returned by the Javascript API. Currently, Tor Browser uses a single, spoofed
User-Agent header for Windows, macOS, and Linux where it advertises the OS as
"Windows NT 10.0" because Windows is the most popular platform. The browser
returns a different UAS when queried by the Javascript API, where it includes
the true OS but not the true OS version number. This functionality is included
in Firefox (thanks to Mozilla's support) and it is used when Firefox's

resistFingerprinting mode is enabled.

- Matt

[Yoav Weiss]

Thanks! That data point seems to support the hypothesis that HTML-level hot-key adaptation may not be necessary, and binding those events with JS later on is a reasonable option.

 

- Matt

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/80c7a29f-9b96-4d61-b531-2744e5562e66%40chromium.org.

[appel...@gmail.com]

Hi Yoav - can you please request a TAG review of this proposal? I think this has enough far-reaching consequences for the web that this is really something the TAG can help with in terms of wide review. Please also see the minutes of our calls this week where we've had some discussions on this topic. Thanks, Dan

On Tuesday, 14 January 2020 11:22:00 UTC, Yoav Weiss wrote:

Interoperability and Compatibility Risk

The compatibility risk varies at different stages.

For the freezing planned for M83, the compatibility risk is low. Existing UA sniffing code will continue to work as expected. It is only future UA sniffing code that will need to change and use the UA client hints instead.

For the unification planned for M85, the compatibility risk is medium. Some sites can modify their responses based on the OS and device model, and those sites will have to change their UA sniffing code to use UA-CH. We expect such sites to be well maintained (otherwise, how can they keep up with OS UI and device model changes?). Therefore, having 4 releases to modify their code seems sufficient.

In the long term, we expect this change to improve compatibility, as UA sniffing based on UA-CH is bound to be more reliable than the current status quo.

As for interoperability, we have other vendors on board with UA freezing, but not necessarily with the UA Client Hints mechanism, that is supposed to replace it. That can create a tricky situation, where developers would need to rely on the User-Agent string for some browsers and on UA-CH for others.

Edge: Public support

Firefox: Public support for freezing the UA string - “freezing the User Agent string without any client hints—seems worth-prototyping”

Safari: Shipped to some extent. Safari has attempted to completely freeze the UA string in the past, without providing an alternative mechanism. That got a lot of pushback, which resulted in somewhat reverting that decision. Nowadays, their UA string seems frozen, other than updates to the OS version and the browser major version.

Alternative implementation suggestion for web developers

For many (most?) uses of UA sniffing today, a better tool for the job would be to use feature detection. Where feature detection fails developers, UA Client Hints are the right path forward.

Potential deployment hurdles compared to status quo:

• Third party services that rely on the UA string would need to convince the sites that include them to delegate that information to them using Feature Policy. 

• The opt-in based mechanism of Client Hints currently suffers from the fact that on the very-first view, the browser have not yet received the opt-in. That means that requiring specific UA information (e.g. device model, platform) on the very-first navigation request may incur a delay. We are exploring options to enable the opt-in to happen further down the stack to avoid that delay. 

[Yoav Weiss]

Hey Daniel! :)

On Thu, Jan 23, 2020 at 2:59 PM <appel...@gmail.com> wrote:

Hi Yoav - can you please request a TAG review of this proposal? I think this has enough far-reaching consequences for the web that this is really something the TAG can help with in terms of wide review. Please also see the minutes of our calls this week where we've had some discussions on this topic. Thanks, Dan

This proposal has already went through TAG review. If there's new TAG input, it might be more efficient to re-open that one?

Also, happy to join one of the TAG calls to discuss this.

[Daniel Appelquist]

With respect, I think a specific proposal to freeze the UA string is different enough in scope from a high level proposal to replace "some headers" with client hints that I think this deserves a new review item. The design is not the issue. The impacts on the rest of the web ecosystem is the issue - and I think this is the part that is making us nervous. 

I am also concerned about how this impacts "minority browsers" - especially as web developers learn what browsers are accessing their sites (and therefore what browsers they need to test on) based on information that may partly be derived from the UA string.

I want to know that you have done the work to understand every way the UA string is being used in the wild and that your proposal accommodates those use cases - especially what some might consider the edge cases such as delivery of content to feature phone browsers.

Dan

[Aaron Tagliaboschi]

On Thu, Jan 23, 2020, 11:29 Daniel Appelquist <appel...@gmail.com> wrote:

With respect, I think a specific proposal to freeze the UA string is different enough in scope from a high level proposal to replace "some headers" with client hints that I think this deserves a new review item. The design is not the issue. The impacts on the rest of the web ecosystem is the issue - and I think this is the part that is making us nervous. 

I am also concerned about how this impacts "minority browsers" - especially as web developers learn what browsers are accessing their sites (and therefore what browsers they need to test on) based on information that may partly be derived from the UA string.

I'd like to point out that, as UA-sniffing has started making unfortunate movements from analysis and logging to feature detection and blacklisting, more and more users and entire browsers are having to lie about who they are and they're unfortunately becoming basically useless for minority browser representation as they have to lie about being other more popular browsers to not get blocks from major sites. See Vivaldi <https://vivaldi.com/blog/user-agent-changes/> and CloudFlare <https://blog.dijit.sh/cloudflare-is-turning-off-the-internet-for-me> for examples.

I want to know that you have done the work to understand every way the UA string is being used in the wild and that your proposal accommodates those use cases - especially what some might consider the edge cases such as delivery of content to feature phone browsers.

I'll let you discuss with Yoav more, but see the Intent to Ship User Agent Client Hints for our proposed replacement <https://groups.google.com/a/chromium.org/d/msg/blink-dev/A4wxFpvqUfA/g7iccl9ICgAJ>

[Yoav Weiss]

On Thu, Jan 23, 2020 at 7:56 PM Daniel Appelquist <appel...@gmail.com> wrote:

With respect, I think a specific proposal to freeze the UA string is different enough in scope from a high level proposal to replace "some headers" with client hints that I think this deserves a new review item. The design is not the issue. The impacts on the rest of the web ecosystem is the issue - and I think this is the part that is making us nervous. 

The design review was rather specific about the fact that it targets the `User-Agent` and the `Accept-Language` headers.

But, if that is somehow helpful, I'm happy to open a new issue.

I am also concerned about how this impacts "minority browsers" - especially as web developers learn what browsers are accessing their sites (and therefore what browsers they need to test on) based on information that may partly be derived from the UA string.

That's indeed a current trade-off for minority browsers - compatibility forces them to often lie in their current User-Agent string (to avoid being blocked), but if they do, developers do not know how many of their users are actually using them.

The UA-CH design is trying to tackle this by enabling browsers to define an easily parsable set of values, that will enable minority browsers to be visible, while claiming they are equivalent to other browsers.

We also want to experiment with applying GREASE to those values - to prevent sites from using allow/block lists in ways that discriminate against browsers.

I want to know that you have done the work to understand every way the UA string is being used in the wild and that your proposal accommodates those use cases - especially what some might consider the edge cases such as delivery of content to feature phone browsers.

Sure. It's important to state that UA-CH does expose the device model (for sites that opt-in) enabling that use-case, among others.

At the same time, we are also discussing ways of enabling that same level of adaptation while exposing less entropy bits to the server.

  

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALiHrgkvpkW3B7ybgYBNj6VD0ugbJvD3vGZ00wOB88hzyyTdOA%40mail.gmail.com.

[jwros...@googlemail.com]

The links provided demostrate Google, Facebook, Cloudflare and Netflix all make use of User-Agents. Not minority businesses.

In one case "faking" the User-Agent resulted in some form of fraud solution kicking in. Not something the majority of web users would ever do.

This proposal is as significant as the Y2K bug but without the clarity of a "fix" for all use cases.Has educating web developers on using the existing solution responsibly been considered? Millions do and would be a lot less impactful.

Given the impact a wider review - including industries like programmatic advertising and publishing that will be dimished financially by the proposal - involving the W3C and other groups as part of wider consultation seems very sensible.

[Yoav Weiss]

On Thu, Jan 23, 2020 at 10:19 PM Yoav Weiss <yo...@yoav.ws> wrote:

On Thu, Jan 23, 2020 at 7:56 PM Daniel Appelquist <appel...@gmail.com> wrote:

With respect, I think a specific proposal to freeze the UA string is different enough in scope from a high level proposal to replace "some headers" with client hints that I think this deserves a new review item. The design is not the issue. The impacts on the rest of the web ecosystem is the issue - and I think this is the part that is making us nervous. 

The design review was rather specific about the fact that it targets the `User-Agent` and the `Accept-Language` headers.

But, if that is somehow helpful, I'm happy to open a new issue.

Opened a dedicated TAG review issue at https://github.com/w3ctag/design-reviews/issues/467

 

[jwros...@googlemail.com]

Dear Yoav,

Thank you for your quick response and the additional information.

Over the past two weeks we have sought evidence to justify this change. We have found none. None of the industry stakeholders we have spoken to were previously aware of the proposal. This means that the usual and necessary protocols of public review are lacking.

In its current form this could easily be interpreted as a partisan gerrymandering attempt by the incumbent dominant player in the field, to the disadvantage of other players.

In our conversations with other players, most recently at the Westminster Policy Forum, we found that they thought that this was all part of a cookie discussion. Solicitation of feedback from a wide global cross section of stakeholders via a neutral party is now required. The W3C comes to mind as fulfilling exactly that role.

This proposal is too radical and has too much potential for disruption to be pushed though quickly, even if you accept the privacy arguments, which we remain unconvinced by.

Considering purely the macro issues associated with good governance and controlled change we observe the following:

1. The proposal has not been reviewed beyond a small group of dedicated, focused and elite engineers who in the main add – not take away – excellent features. This change impacts global industry sectors as diverse as publishing, marketing, advertising, technology, charities and more in yet undetermined ways. The risk is equivalent to the Y2K bug.

2. Online platforms – and Google in particular – are the subject of a wide ranging Competition and Market Authority (CMA) review covering the subjects of this proposal. The full report is expected to be published on 2nd July 2020. An interim report was published on 18th December 2019. The interim report establishes a balance between the needs of individuals for privacy and for markets and technology to function efficiently. Its conclusions should inform this proposal.

3. This proposal will disproportionally benefit Google as in practice it will remove data for smaller platform operators and millions of others. Paragraph 60 appendix E of the CMA review states.

“Google is the platform with the largest dataset collected from its leading consumer-facing services such as YouTube, Google Maps, Gmail, Android, Google Chrome and from partner sites using Google pixel tags, analytical and advertising services. A Google internal document recognises this advantage saying that ‘Google has more data, of more types, from more sources than anyone else’.”

The appendix includes the following diagram to illustrate the point.

 

4. The CMA are yet to comment on Google’s role in relation to influence over web standards via the Chromium project and other means. Microsoft’s decision to adopt Chromium, and the apparent decline of Firefox are likely to be topics they comment on in July 2020.

5. We need more time to discuss the impact with our users. We believe this is true for others. 

As just one example the AdCom specification needs to be updated. Only once this is done can all publishers, SSPs, exchanges and DSPs adopt the new schema. If any of these parties do not make the modifications all are disadvantaged. The change needs to be made in lockstep.

Many trade bodies and organisations are focused on the implications associated with the publicity concerning 3rd party cookies. They are only just becoming aware of this proposal. We are encouraging them to engage publicly but respect the demands on their time, limited resources and the sensitivities concerning the topic of privacy.

There are many more arguments concerning assumptions, insufficient evidence, implementation, control over privacy (who decides?), and the technical impacts of the proposal yet to be resolved. 

In summary, this is a change that requires careful and widespread consideration, and a significant effort to socialise for it to be recognised by all as legitimately in the public interest. Without mature reflection and appropriate implementation delay it will be perceived as market manipulation by the incumbent player.

Regards,

James Rosewell - for self and 51Degrees

[mh04...@gmail.com]

Hi Yoav,

First and foremost, thank you for giving the opportunity to members of the community to engage in this discussion.

I'm very concerned regarding the reasoning behind this change: bits of entropy.

As a small ad network, we use ip and user agent data to combat ad fraud. These same bits of entropy are used when we detect ad fraud. This is virtually impossible to do if all we are getting is a rotating vpn-based ip address and "Chrome 74". At best, we have to wait for another request to get the rest of the UA data (significantly reducing our ad serving speed) or worst case, we will "exceed the user's privacy budget" and be denied this information altogether.

Who decides how "the user agent can make reasonable decisions about when to honor requests for detailed user agent hints"? There is absolutely no doubt that your own properties will be ranked high in a hypothetical "trust/privacy" rating. There is nothing stopping you or your successors from abusing this power against smaller players like in the case with Yelp.

Just because your organization has hundreds of millions of logged in active users across your various web properties and devices (search, chrome, android, chrome os, youtube, gmail, pixel etc.), it is significantly easier to run ad fraud analysis and protect your own ad network as the rest of us bite the dust.

On the github repo it states "Top-level sites a user visits frequently (or installs!) might get more granular data than cross-origin, nested sites, for example". What about smaller sites just starting out? With all the large players (that already have top-level sites a user visits frequently) remaining untouched, you are effectively crippling competition from smaller players.

Vast majority of the internet users simply don't care about this change and the handful that do, are probably underestimating the anti-trust issues that this change brings. You have to be naive to think that there is absolutely no conflict of interest when a company that collects the most amount of data on earth is limiting what other, less frequently visited sites are allowed to see?

Reducing "bits of entropy" is simply not a good enough reason to proceed with the change. I adore chrome and personally use it every day, however, I'd like to point out to the community that this change is not in everyone's best interest.

TL;DR: We need the full os and browser version to survive as a small ad network. And more importantly, we need this data as part of every first http request, without being discriminated against for being a less frequently visited / smaller website.

Regards,

Andy

[davet...@gmail.com]

Firstly, thank you to Mike West for all the security enhancements and Yoav for the work on preload.

After watching the video by Vivaldi browser, posted by Yoav, I'm shocked and extremely disappointed in Google (yet again!).

Let's look at the Vivaldi browser example:

They have their user-agent saying "Vivaldi browser" and they get blocked by Google and Google services tsk tsk!

Mike West creates User Agent Client Hints in 2018 according to this paper: https://tools.ietf.org/html/draft-west-ua-client-hints-00

Two years pass and Google want to deprecate and freeze user-agents and switch over to User Agent Client Hints.

So running some tests with this spec we get the following results:

sec-ch-ua: Vivaldi 79
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.131 Safari/537.36

What's there to stop Google and Google services from blocking "Vivaldi browser" using:

 

sec-ch-ua: Vivaldi 79

NOTHING!

It clearly seems that this spec has not addressed the actually issues at hand.

I'm so disappointed in Google's behaviour that I sent that video for the European Ombudsman (a competition watchdog) to look into that case and hopefully fine Google, for doing such shady practises!

Another thing which is totally wrong can be seen here: https://www.chromestatus.com/feature/5704553745874944

It lists only 4 major browser vendors: Chrome, Firefox, Edge and Safari.

Let's rename this to the "Browser Cartel"

Can organisations such as W3C, ICANN, WIGC, WHATWG etc. all start doing things differently.

For example with this spec, should not get a green light until most of the browser vendors have given feedback.

Here's a list of the most popular browsers (something I feel Google wants to hide from the public):

• 115 Browser
• 1stBrowser
• 2345 Browser
• 2345 Chrome
• 2345 Explorer
• 360 Secure Browser
• 360 Speed Browser
• 37abc
• 7 Star
• ABrowse
• Acoo Browser
• Aloha Browser
• Amigo
• Android
• Android WebView
• AOL Desktop
• AOL Shield
• APUSBrowser
• Arora
• Avant
• Avast SafeZone
• Avira Scout
• Baidu Browser
• Baidu Browser HD
• Beamrise
• BlackBerry
• Blackberry Playbook Tablet
• Browzar
• Camino
• Cent Browser
• Chedot
• Cheshire
• Chimera
• Chrome
• ChromePlus
• Chromium
• CM Browser
• Coc Coc Browser
• CometBird
• CoolNovo
• Crazy Browser
• Cunaguaro
• Cyberfox
• Diigo Browser
• Dillo
• DoCoMo
• Dolfin
• Dorado WAP Browser
• Dragon
• Ecosia
• Edge
• Edge Mobile
• ELinks
• Epiphany
• EudoraWeb
• Fennec
• Firebird
• Firefox
• Firefox Focus
• Firefox for iOS
• Flock
• FlyFlow
• Freebox Browser
• Galeon
• Google Desktop
• Headless Chrome
• IBrowse
• iCab
• Iceape
• IceCat
• Icedove
• IceDragon
• Iceweasel
• IE
• IEMobile
• iLunascape
• Iridium Browser
• Iron
• Iron Mobile
• Jasmine
• jBrowser
• JUZI Browser
• K-Meleon
• Kazehakase
• Kindle
• Kindle Fire
• Kinza
• KKMAN
• Klondike
• Konqueror
• liebao
• Light
• Links
• lolifox
• Lovense Browser
• luakit
• Lunascape
• Lynx
• MAUI Wap Browser
• Maxthon
• Maxthon Nitro
• Mercury
• MicroB
• Midori
• Minimo
• Miui Browser
• Mobicip
• Mobile Safari UIWebView
• MobileIron
• Model S Browser
• Mozilla
• MultiZilla
• Navigator
• NetFront
• NetFront NX
• Netscape
• Nichrome
• Nintendo Browser
• Nokia Browser
• Nokia Proxy Browser
• Obigo Q
• Oculus Browser
• OmniWeb
• OneBrowser
• open-webkit-sharp
• Openwave Mobile Browser
• Opera
• Opera Mini
• Opera Mobile
• Opera Neon
• Oppo Browser
• Oregano
• Origin
• Otter
• PaleMoon
• Phantom Browser
• PhantomJS
• Phoenix
• Playstation Browser
• Puffin
• Qiyu Browser
• QQBrowser
• Qt
• QtWebEngine
• QuickLook
• QupZilla
• rekonq
• RockMelt
• Safari
• Sailfish Browser
• Samsung
• Samsung Browser
• Samsung CrossApp
• Samsung WebView
• SeaMonkey
• SEMC
• Seznam Browser
• Silk
• Sleipnir
• SlimBoat
• SlimBrowser
• SlimerJS
• SlimJet Browser
• SmartTV WebBrowser
• SmartViera
• Sogou Explorer
• Sogou Mobile
• Sony PSP
• Sputnik Browser
• Sraf
• Stainless
• Start Browser
• Sunrise
• SuperBird
• Surf Browser
• Swing
• Teleca-Obigo
• TenFourFox
• Tenta Browser
• TeslaBrowser
• TheWorld
• Tizen Browser
• UC Browser
• UC Browser Mini
• UR Browser
• Vivaldi
• Vivo Browser
• VmWare Browser
• w3m
• Waterfox
• WebKit/webOS
• WeTab Browser
• Whale Browser
• WhiteHat Aviator
• WinWAP
• WKBrowser
• Wyzo
• YaaniBrowser
• Yandex Browser
• Zvu

[ka...@smarterclick.co.uk]

NO

On Tuesday, 14 January 2020 11:22:00 UTC, Yoav Weiss wrote:

Contact emails

yoav...@chromium.org, aaro...@chromium.org

[Gil Meroz]

If I have 2 servers on 2 different sub domains, and there's a request to one of the domains that return the UA information request header, will Chrome add the Client Hints to the next request on the other domain?

[Gil Meroz]

Will Chrome automatically add the Client Hints on the request header on redirected requests (HTTP status 302) ?

I mean when the first HTTP response has the `Location` header and the UA information request headers. In such case, when the browser automatically launch the second request, will it include the Clients Hints ?

[Yoav Weiss]

On Sun, Feb 9, 2020 at 10:02 PM 'Gil Meroz' via blink-dev <blin...@chromium.org> wrote:

If I have 2 servers on 2 different sub domains, and there's a request to one of the domains that return the UA information request header, will Chrome add the Client Hints to the next request on the other domain?

The Accept-CH opt-in needs to be delivered over top-level navigation responses in order to be applied: https://github.com/WICG/client-hints-infrastructure#accept-ch

The opt-in is per origin, so 2 different sub-domains will not be impacted by each other's opt-ins. 

[Yoav Weiss]

On Sun, Feb 9, 2020 at 10:02 PM 'Gil Meroz' via blink-dev <blin...@chromium.org> wrote:

Will Chrome automatically add the Client Hints on the request header on redirected requests (HTTP status 302) ?

I mean when the first HTTP response has the `Location` header and the UA information request headers. In such case, when the browser automatically launch the second request, will it include the Clients Hints ?

If the redirect is cross-origin, the opt-in from the first origin will not be applied to the second origin.

If the redirect is same-origin, the opt-in should apply to the second request. 

[Yoav Weiss]

Hey all!

As this intent relies on UA Client Hints and as the UA-CH intent is on hold due to great feedback on the proposal, I wanted to update this thread regarding timelines.

Originally, we planned to ship UA-CH for M81, but as feedback resulted in proposal changes, that didn't happen.

I'm currently focused on specifying and implementing those changes and then shipping UA-CH. The timelines for this intent are likely to be derived from the time in which that UA-CH would actually ship. 

As a result, the timelines mentioned in the OP are likely to slip as well. I'll update this thread once we have line of sight as to what those new timelines would be.

Cheers :)

Yoav

[dat...@verizonmedia.com]

Yoav,

 

Thanks for the notice.

Even a UA string may contain some fingerprinting signals that can be mis-used to against user privacy, it also contains signals that are necessary to customize web pages for publishers and to fight ad frauds. The privacy concerns here are really case by case. Related institutes should work on clear regulations on "UA string usage agreements" for all players. Physically re-shape the UA string in a single browser triggers significant changes at many parties and may not address the actual privacy concerns. 

Datong

 

[andrea...@googlemail.com]

I have the same problem. If the first request ('https://my.domain/url') will response with a 302 redirect and the target is a local ('/url/without/protocol/and/domain') or a full ('https://my.domain/otherurl') location and the 302 response includes the 'accept-ch' header the second request will not have any UA client hints.

I'm testing with Chrome 78.0.3904.70.

What means same-origin exactly?

[thorsten...@sap.com]

Hi Yoav,

 

could you please provide an update regarding time line of both projects.

1) Implementing UA Client Hints

2) Freezing and Removing user agent string

 

Could you please make an example of the freeze user agent string. It is very important to decide on the criticality of the change.

 

Please be aware that we would need enough time between 1) and 2) to implement and ship and for customers to implement the fixes. Especially on-prem customer running mission critical systems needs at least one year to get the systems updated. A time plan like initially told within half a year is not realistic for business customers running onprem systems.

 

Thanks Thorsten 

[steal...@gmail.com]

What does unify mean? Does it mean that existing user agents will be changed?

E.g. A user agent could have the string: "iPhone OS 11_3_1". When unification happens, will it change the string to "iPhone OS"?

[ad...@atlantacounselingsolutions.com]

Given the impact of Coronavirus on people, companies, people, budget, people, timelines, people, feasibility, people, 3rd party impact, everything else listed in this post above me:

Can we move this out a year?

Bret Lowery

[Samridh Saluja]

Andrea,

I am looking to test the UA Client hints in my environment to ensure compatibility. Could you share where the relevant information regarding testing can be found? Thanks

[Yoav Weiss]

Hey all,

As part of our close monitoring of the COVID-19 impact on the web ecosystem and our recent announcements about changes to Chrome’s upcoming release schedule, I wanted to share an update on the upcoming User-Agent Client Hints project.

First, the phase of the project that reduces the amount of available information in the traditional User-Agent string to browser (e.g. “Chrome”), major browser version, and mobile/desktop, has been deferred until at least 2021 to provide additional time for the ecosystem to evaluate the new UA-CH capabilities.

Second, the initial phase of the UA-CH project which introduces new Client Hints focused on providing similar informational capabilities to the traditional User-Agent HTTP header string, is now expected to be ready to ship for Chrome version 84 which is scheduled to be released to stable on July 14th. This release is intended to allow for developers to experiment and provide feedback. This introduction is entirely additive, with no expected impact on existing systems. 

While work on UA-CH continues, we don’t currently know in what ways or for how long COVID-19 will impact the web ecosystem’s ability to test and implement support for this change. We will continue to monitor the situation and revisit this topic as the situation evolves.

Thanks,

Yoav

[mi...@integralads.com]

Hi Yoav,

Is there an updated timeline anywhere? What about any documentation of responses to the challenges to the plan raised in the thread? Is there a better place than this thread to follow the developments?
--
This message (including any attachments) may contain confidential,
proprietary, private and/or privileged information. The information is
intended to be for the use of the individual or entity designated above. If
you are not the intended recipient of this message, please notify the
sender immediately, and delete the message and any attachments. Any
disclosure, reproduction, distribution or other use of this message or any
attachments by an individual or entity other than the intended recipient is
strictly prohibited. In addition, emails sent from and to this
integralads.com <http://integralads.com>domain are monitored, archived,
and subject to disclosure, including in connection with regulatory or other
legal processes.

[Yoav Weiss]

On Sat, Jun 6, 2020 at 6:45 AM micah via blink-dev <blin...@chromium.org> wrote:

Hi Yoav, 

Is there an updated timeline anywhere?

No updated timeline beyond my previous message here. I'll update this thread once we know more.

 

What about any documentation of responses to the challenges to the plan raised in the thread?

Any specific issue you feel wasn't addressed?

 

Is there a better place than this thread to follow the developments?

--
This message (including any attachments) may contain confidential,
proprietary, private and/or privileged information. The information is
intended to be for the use of the individual or entity designated above. If
you are not the intended recipient of this message, please notify the
sender immediately, and delete the message and any attachments. Any
disclosure, reproduction, distribution or other use of this message or any
attachments by an individual or entity other than the intended recipient is
strictly prohibited. In addition, emails sent from and to this
integralads.com  <http://integralads.com>domain are monitored, archived,
and subject to disclosure, including in connection with regulatory or other
legal processes.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/17fc95fa-60c7-46f1-b294-39de80aac074o%40chromium.org.

[mi...@integralads.com]

Any specific issue you feel wasn't addressed?

Not exactly. I've only been following this project tangentially. Everywhere I look I see long lists of issues and discussions that I'm not eager to try to digest in their entirety :). Our limited testing in April showed a few differences between the spec and the implementation at that point and raised a bunch of questions for how we'll adapt to this change. Also, some of the related blog posts, such as DeviceAtlas's, seem to raise significant questions. I'm just trying to find a definitive place from which to watch this project and to get a better understanding of what exactly is going to be shipped when.

---

This message (including any attachments) may contain confidential, proprietary, private and/or privileged information. The information is intended to be for the use of the individual or entity designated above. If you are not the intended recipient of this message, please notify the sender immediately, and delete the message and any attachments. Any disclosure, reproduction, distribution or other use of this message or any attachments by an individual or entity other than the intended recipient is strictly prohibited. In addition, emails sent from and to this integralads.com domain are monitored, archived, and subject to disclosure, including in connection with regulatory or other legal processes.

[Aaron Tagliaboschi]

> Our limited testing in April showed a few differences between the spec and the implementation at that point.

We've done quite a bit of work on the internals of client hints in the past few months; if you can recall, do you remember anything specific?

--
You received this message because you are subscribed to a topic in the Google Groups "blink-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/blink-dev/-2JIRNMWJ7s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/b1ada7d1-afc7-4474-aafe-f1a0a5d01bdeo%40chromium.org.

[Aaron Tagliaboschi]

On Thu, Jun 11, 2020, 08:30 KOSUI IWASA <iwasa...@gmail.com> wrote:

Because Chrome for iOS uses Webkit, is it out-of-topic that if the UA String will be frozen in Chrome for iOS or not?

Yes, deprication on iOS will depend on User Agent Client Hint support on from WebKit, so it will have its own timeline