Intent to deprecate remove: Sending user-agent to HTTP proxies when establishing tunnels.
Matt Menke
Primary eng (and PM) emails
Summary
When establishing an HTTPS connection over an HTTP/HTTPS/H2/QUIC proxy, we send a CONNECT request with its own set of headers. Currently, if the original HTTPS request had a custom user-agent string, we would send that with the CONNECT request, rather than the default user-agent header. I plan to remove this behavior, and just use the default global user-agent header.
Motivation
We pool sockets to the same "group" together (loosely defined as origin + privacy mode + proxy triplet). Since that triplet doesn't include user-agent, we can end up servicing a request with a pre-existing HTTPS tunnel established using a user-agent other than the one associated with the original request.
Moreover, we only bind sockets to socket requests after we've established a tunnel, so that we can if there are two simultaneous requests with different user-agents, and we create new connections for each request, it's possible that 2, 1, or neither request will end up with a connection creating using the user agent associated with that request.
Preconnects further muddy the waters, as there's no way to specify user-agent headers for them. They just always use the network service's global default user-agent string.
Beyond that, there are other parameters that have similar behavior to user agent (Like what client certs we use) that have gotten us into serious trouble. I want to remove all socket creation parameters that can vary within a "group" so we can just have one set of parameters per group, and avoid this sort of issue in the future. User-agent itself is relatively benign, but getting rid of it helps rectify a problematic design decision.
More generally, behavior that "kind of" works is problematic, and we can't really fix things here without either further splitting the socket pools in a way a website can control directly, or closing sockets when there's a user-agent mismatch, which also seems sub-optimal.
Interoperability and Compatibility Risk
In order for this to cause compatibility issues, either a user/extension/network-configured proxy would have to coordinate with a site/devtools/extension-configured user agent string, and have different behaviors based on it. While that doesn't make compatibility issues impossible, it seems to make them fairly improbable.
I have not tested other browsers behaviors here, though neither the fetch spec nor the HTTP CONNECT spec define what the behavior should be here.
Edge: ???
Firefox: ???
Safari: ???
Alternative implementation suggestion for web developers
It's primarily proxy developers that this affects, not web developers, since while web developers can set user-agent strings, they can't see what user-agent string was sent to the proxy. Proxies really shouldn't be depending on user-agent string. It's easy to set, and not verifiable (And it's easy to use the wrong one, as noted above, particularly given Chrome's auto-preconnect behavior). The behavior is pretty broken, as noted above, so depending on this behavior doesn't really work.
If proxy developers want some way to verify the identity of someone using their proxy, HTTP auth is the recommended mechanism.
I believe we also support client certs for HTTP proxies, though I'm not positive of that.
Usage information from UseCounter
We can't really get useful information on this here - blink generally overrides user-agent on all requests by default, I believe. This is also a fairly small removal, so I don't think it's worth gathering data on it.
Entry on the feature dashboard
I'm not sure if this really counts as a web-facing change. It's a pretty minor change, and I'm not expecting much fallout.
Requesting approval to remove too?
Yes - though this is a sufficiently small change, current behavior is sufficiently broken, and it's sufficiently non-web-facing, that I'm not sure it really needs approval. I just thought it would be nice to announce somewhere that I'm making a change, and give people a chance to push back on that (Or push back on me sending this email in the first place, given how minor a change it is). I admittedly broke some sites in M72 and didn't announce things, so am opting to be more cautious this time around.
Joe Medley
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEK7mvrprvHsZOFZXG3fVNu2Ccr76bCk7EX91nO3BT%3Da5Z%2BUow%40mail.gmail.com.
Matt Menke
Torne (Richard Coles)
Primary eng (and PM) emails
Summary
When establishing an HTTPS connection over an HTTP/HTTPS/H2/QUIC proxy, we send a CONNECT request with its own set of headers. Currently, if the original HTTPS request had a custom user-agent string, we would send that with the CONNECT request, rather than the default user-agent header. I plan to remove this behavior, and just use the default global user-agent header.
Motivation
We pool sockets to the same "group" together (loosely defined as origin + privacy mode + proxy triplet). Since that triplet doesn't include user-agent, we can end up servicing a request with a pre-existing HTTPS tunnel established using a user-agent other than the one associated with the original request.
Moreover, we only bind sockets to socket requests after we've established a tunnel, so that we can if there are two simultaneous requests with different user-agents, and we create new connections for each request, it's possible that 2, 1, or neither request will end up with a connection creating using the user agent associated with that request.
Preconnects further muddy the waters, as there's no way to specify user-agent headers for them. They just always use the network service's global default user-agent string.
Beyond that, there are other parameters that have similar behavior to user agent (Like what client certs we use) that have gotten us into serious trouble. I want to remove all socket creation parameters that can vary within a "group" so we can just have one set of parameters per group, and avoid this sort of issue in the future. User-agent itself is relatively benign, but getting rid of it helps rectify a problematic design decision.
More generally, behavior that "kind of" works is problematic, and we can't really fix things here without either further splitting the socket pools in a way a website can control directly, or closing sockets when there's a user-agent mismatch, which also seems sub-optimal.
Interoperability and Compatibility Risk
In order for this to cause compatibility issues, either a user/extension/network-configured proxy would have to coordinate with a site/devtools/extension-configured user agent string, and have different behaviors based on it. While that doesn't make compatibility issues impossible, it seems to make them fairly improbable.
I have not tested other browsers behaviors here, though neither the fetch spec nor the HTTP CONNECT spec define what the behavior should be here.
Edge: ???
Firefox: ???
Safari: ???
Alternative implementation suggestion for web developers
It's primarily proxy developers that this affects, not web developers, since while web developers can set user-agent strings, they can't see what user-agent string was sent to the proxy. Proxies really shouldn't be depending on user-agent string. It's easy to set, and not verifiable (And it's easy to use the wrong one, as noted above, particularly given Chrome's auto-preconnect behavior). The behavior is pretty broken, as noted above, so depending on this behavior doesn't really work.
If proxy developers want some way to verify the identity of someone using their proxy, HTTP auth is the recommended mechanism.
I believe we also support client certs for HTTP proxies, though I'm not positive of that.
Usage information from UseCounter
We can't really get useful information on this here - blink generally overrides user-agent on all requests by default, I believe. This is also a fairly small removal, so I don't think it's worth gathering data on it.
Entry on the feature dashboard
I'm not sure if this really counts as a web-facing change. It's a pretty minor change, and I'm not expecting much fallout.
Requesting approval to remove too?
Yes - though this is a sufficiently small change, current behavior is sufficiently broken, and it's sufficiently non-web-facing, that I'm not sure it really needs approval. I just thought it would be nice to announce somewhere that I'm making a change, and give people a chance to push back on that (Or push back on me sending this email in the first place, given how minor a change it is). I admittedly broke some sites in M72 and didn't announce things, so am opting to be more cautious this time around.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
Matt Menke
On Wed, 20 Feb 2019 at 22:17 'Matt Menke' via blink-dev <blin...@chromium.org> wrote:Primary eng (and PM) emails
Summary
When establishing an HTTPS connection over an HTTP/HTTPS/H2/QUIC proxy, we send a CONNECT request with its own set of headers. Currently, if the original HTTPS request had a custom user-agent string, we would send that with the CONNECT request, rather than the default user-agent header. I plan to remove this behavior, and just use the default global user-agent header.
Motivation
We pool sockets to the same "group" together (loosely defined as origin + privacy mode + proxy triplet). Since that triplet doesn't include user-agent, we can end up servicing a request with a pre-existing HTTPS tunnel established using a user-agent other than the one associated with the original request.
Moreover, we only bind sockets to socket requests after we've established a tunnel, so that we can if there are two simultaneous requests with different user-agents, and we create new connections for each request, it's possible that 2, 1, or neither request will end up with a connection creating using the user agent associated with that request.
Preconnects further muddy the waters, as there's no way to specify user-agent headers for them. They just always use the network service's global default user-agent string.
Beyond that, there are other parameters that have similar behavior to user agent (Like what client certs we use) that have gotten us into serious trouble. I want to remove all socket creation parameters that can vary within a "group" so we can just have one set of parameters per group, and avoid this sort of issue in the future. User-agent itself is relatively benign, but getting rid of it helps rectify a problematic design decision.
More generally, behavior that "kind of" works is problematic, and we can't really fix things here without either further splitting the socket pools in a way a website can control directly, or closing sockets when there's a user-agent mismatch, which also seems sub-optimal.
Interoperability and Compatibility Risk
In order for this to cause compatibility issues, either a user/extension/network-configured proxy would have to coordinate with a site/devtools/extension-configured user agent string, and have different behaviors based on it. While that doesn't make compatibility issues impossible, it seems to make them fairly improbable.
Android WebVIew supports the embedding app overriding the useragent string and this is commonly used in apps. It's conceivable (though I don't know of an example) that some proxy is relying on it..
Torne (Richard Coles)
Matt Menke
Joe Medley
David Benjamin
Primary eng (and PM) emails
Summary
When establishing an HTTPS connection over an HTTP/HTTPS/H2/QUIC proxy, we send a CONNECT request with its own set of headers. Currently, if the original HTTPS request had a custom user-agent string, we would send that with the CONNECT request, rather than the default user-agent header. I plan to remove this behavior, and just use the default global user-agent header.
Motivation
We pool sockets to the same "group" together (loosely defined as origin + privacy mode + proxy triplet). Since that triplet doesn't include user-agent, we can end up servicing a request with a pre-existing HTTPS tunnel established using a user-agent other than the one associated with the original request.
Moreover, we only bind sockets to socket requests after we've established a tunnel, so that we can if there are two simultaneous requests with different user-agents, and we create new connections for each request, it's possible that 2, 1, or neither request will end up with a connection creating using the user agent associated with that request.
Preconnects further muddy the waters, as there's no way to specify user-agent headers for them. They just always use the network service's global default user-agent string.
Beyond that, there are other parameters that have similar behavior to user agent (Like what client certs we use) that have gotten us into serious trouble. I want to remove all socket creation parameters that can vary within a "group" so we can just have one set of parameters per group, and avoid this sort of issue in the future. User-agent itself is relatively benign, but getting rid of it helps rectify a problematic design decision.
More generally, behavior that "kind of" works is problematic, and we can't really fix things here without either further splitting the socket pools in a way a website can control directly, or closing sockets when there's a user-agent mismatch, which also seems sub-optimal.
Interoperability and Compatibility Risk
In order for this to cause compatibility issues, either a user/extension/network-configured proxy would have to coordinate with a site/devtools/extension-configured user agent string, and have different behaviors based on it. While that doesn't make compatibility issues impossible, it seems to make them fairly improbable.
I have not tested other browsers behaviors here, though neither the fetch spec nor the HTTP CONNECT spec define what the behavior should be here.
Edge: ???
Firefox: ???
Safari: ???
Alternative implementation suggestion for web developers
It's primarily proxy developers that this affects, not web developers, since while web developers can set user-agent strings, they can't see what user-agent string was sent to the proxy. Proxies really shouldn't be depending on user-agent string. It's easy to set, and not verifiable (And it's easy to use the wrong one, as noted above, particularly given Chrome's auto-preconnect behavior). The behavior is pretty broken, as noted above, so depending on this behavior doesn't really work.
If proxy developers want some way to verify the identity of someone using their proxy, HTTP auth is the recommended mechanism.
I believe we also support client certs for HTTP proxies, though I'm not positive of that.
Usage information from UseCounter
We can't really get useful information on this here - blink generally overrides user-agent on all requests by default, I believe. This is also a fairly small removal, so I don't think it's worth gathering data on it.
Entry on the feature dashboard
I'm not sure if this really counts as a web-facing change. It's a pretty minor change, and I'm not expecting much fallout.
Requesting approval to remove too?
Yes - though this is a sufficiently small change, current behavior is sufficiently broken, and it's sufficiently non-web-facing, that I'm not sure it really needs approval. I just thought it would be nice to announce somewhere that I'm making a change, and give people a chance to push back on that (Or push back on me sending this email in the first place, given how minor a change it is). I admittedly broke some sites in M72 and didn't announce things, so am opting to be more cautious this time around.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
Matt Menke
(I assume you meant "Sending *custom* user-agent to HTTP proxies" in the subject?)
On Wed, Feb 20, 2019 at 9:17 PM 'Matt Menke' via blink-dev <blin...@chromium.org> wrote:Primary eng (and PM) emails
Summary
When establishing an HTTPS connection over an HTTP/HTTPS/H2/QUIC proxy, we send a CONNECT request with its own set of headers. Currently, if the original HTTPS request had a custom user-agent string, we would send that with the CONNECT request, rather than the default user-agent header. I plan to remove this behavior, and just use the default global user-agent header.
Motivation
We pool sockets to the same "group" together (loosely defined as origin + privacy mode + proxy triplet). Since that triplet doesn't include user-agent, we can end up servicing a request with a pre-existing HTTPS tunnel established using a user-agent other than the one associated with the original request.
Moreover, we only bind sockets to socket requests after we've established a tunnel, so that we can if there are two simultaneous requests with different user-agents, and we create new connections for each request, it's possible that 2, 1, or neither request will end up with a connection creating using the user agent associated with that request.
Preconnects further muddy the waters, as there's no way to specify user-agent headers for them. They just always use the network service's global default user-agent string.
If it hasn't been working for preconnects, I suspect this hasn't even been "kind of" working all this time.
(Do we preserve this custom UA string on cross-site redirects? If not, that would also be a reason not to send it to the proxy. That would mean that, but for the proxy case, someone could reliably assume that a custom UA string for an HTTPS host will only get sent to that host. The point of CONNECT is that the proxy is not trusted to speak for the host and so it should not get to see that UA string.)
Torne (Richard Coles)
Yea, I don't think we can make that work (And it's already broken). If an app doesn't make any other network requests, never preconnects (And doesn't come across a site that uses preconnects, either), and always overrides it with the same value, it will get consistent behavior, but in general, it just doesn't work.
This does only affect the user agent string sent to proxies, but an app where you can control both the proxy and the user agent could theoretically be using user-agent as a poor-man's proxy authentication.
Matt Menke
On Thu, 21 Feb 2019 at 11:31 Matt Menke <mme...@google.com> wrote:Yea, I don't think we can make that work (And it's already broken). If an app doesn't make any other network requests, never preconnects (And doesn't come across a site that uses preconnects, either), and always overrides it with the same value, it will get consistent behavior, but in general, it just doesn't work.Apps with just one webview that happen not to load anything that triggers preconnects, service worker usage, etc will probably work at the moment by accident, yeah.This does only affect the user agent string sent to proxies, but an app where you can control both the proxy and the user agent could theoretically be using user-agent as a poor-man's proxy authentication....and apps that are doing something explicitly using a proxy where they control both sides might well be the ones that are doing it in the case that mostly works. :(I'm not trying to block this - it seems like a good idea given that it doesn't work reliably now - just making you aware that this is a potential compatibility risk that we will need to watch out for.
Torne (Richard Coles)
On Thu, Feb 21, 2019 at 11:39 AM Torne (Richard Coles) <to...@chromium.org> wrote:On Thu, 21 Feb 2019 at 11:31 Matt Menke <mme...@google.com> wrote:Yea, I don't think we can make that work (And it's already broken). If an app doesn't make any other network requests, never preconnects (And doesn't come across a site that uses preconnects, either), and always overrides it with the same value, it will get consistent behavior, but in general, it just doesn't work.Apps with just one webview that happen not to load anything that triggers preconnects, service worker usage, etc will probably work at the moment by accident, yeah.This does only affect the user agent string sent to proxies, but an app where you can control both the proxy and the user agent could theoretically be using user-agent as a poor-man's proxy authentication....and apps that are doing something explicitly using a proxy where they control both sides might well be the ones that are doing it in the case that mostly works. :(I'm not trying to block this - it seems like a good idea given that it doesn't work reliably now - just making you aware that this is a potential compatibility risk that we will need to watch out for.I appreciate it, and am glad you brought it up. Is there some place we can announce this as a potential compatibility breaking change to WebView embedders, and suggest workarounds?
It looks like we don't give WebView embedders access to set the entire global UA string - https://cs.chromium.org/chromium/src/android_webview/browser/aw_content_browser_client.cc?type=cs&g=0&l=240
PhistucK
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEV-rjf0ZRDs42Gqzz7vJudvb2isoqsv2DzpoEb0M2F43Av9Aw%40mail.gmail.com.
David Benjamin
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BXJapNhKXGDCbh0uc5cR-JF%3DpLqeqbLUoWYK8jDd_Q%3Dw%40mail.gmail.com.
Daniel Bratell
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAF8qwaAHHWxJ3DNUZosm5pRUT6xucM8fuBHUNRO4CS%2B4HQm-8g%40mail.gmail.com.
Matt Menke
I have some questions, based on my assumptions after reading this thread:Assumptions-------------------------* There is a compatibility risk here but we have no way of quantifying it with metrics or source code analysis since dependencies will be hidden in server side code.* Leaving it as it is will leave an unpredictable race condition if more than one user agent string is used by a client, and especially so if there is more than one per-request user agent string.* Most use cases probably do not have more than one custom user agent string so mostly things will "work" because they will always use the same custom user agent string or no custom user agent string.
* More use of pooling and preconnects will increase the race condition exposure, alternatively the pools will have to be made "per user agent" which limits their usefulness (how much?).
* There is no "deprecation" but just a removal of the ability to customize proxy connect user agents.
* Doing a change and carefully monitoring the feedback (and possibly revert) might be the only way to be sure it's not breaking too much.Questions------------------The intent writes that "neither the fetch spec nor the HTTP CONNECT spec define what the behavior should be here.". What actually is "here" and what do the specs define? Can we patch them to be well defined?
Do other browsers send user agents for CONNECT normally? Is it just for the "per request custom user agent" we don't know what they do?
What is the testing situation like? Can this be tested? Is it tested?
Matt Menke
Questions------------------The intent writes that "neither the fetch spec nor the HTTP CONNECT spec define what the behavior should be here.". What actually is "here" and what do the specs define? Can we patch them to be well defined?"Here" is defined as when establishing an SSL tunnel through an HTTP proxy, it is unclear if a website-configured User-Agent header should be send with the CONNECT request. There are also the related questions of when devtools sets the header, and when extensions set the header.
Philip Jägenstedt
under user control and not observable to the web developer unless the
proxy and real server are sharing information, I would say this isn't
web exposed and doesn't affect web developers, at least not to a
degree that warrants going through the intent process.
There is some risk, perhaps especially with WebView setups, but I
think the trade-offs and decisions should be made by the product/code
owners, not Blink API owners. That being said, the justification for
the change makes sense to me, and it seems low risk enough to try.
> You received this message because you are subscribed to the Google Groups "blink-dev" group.
Daniel Bratell
Blink API Owners scope. From my point of view, feel free to move ahead.
/Daniel