Possible bug in Chrome's HSTS - help me confirm?

[Dan L]

(Cross-posted from chromium-discuss)

Greetings,

In our corporate environment, a web proxy sits between users and the internet. The proxy is transparent, and so enforces authentication by spoofing the original content server, and redirecting unauthenticated web requests with a 302 to a URL of my choice, in this case, to the host "proxysg" for HTTP requests, and "https://proxysg:4443" for HTTPS requests. Some Chrome users have recently reported an error when accessing some web sites. The error page reads: "proxysg refused to connect". The URL in the address bar when this appears is a variation on the authentication URL, like so: https://proxysg/. Note that it's HTTPS, but not using port 4443. This only seems to happen when accessing a plain HTTP web site. Since our proxy doesn't have a listener on 443, a RST is returned when the browser attempts to connect to that address, resulting in the error.

For affected users, I find an entry for the host name "proxysg" when queried under chrome://net-internals/#hsts. The problem goes away when the entry is deleted.

Our proxy doesn't issue Strict-transport-security headers, so I don't understand how it could have found its way into chrome's HSTS cache. I'm unable to replicate the problem, so I don't know under what conditions Chrome will add that host to HSTS. We have relatively few Chrome users, so the affected environment is limited so far to Windows 7, and Chrome v58.

I'm trying to get a better understanding of this issue in order to be able to replicate it in a controlled setting. Has anyone in a similar environment seen this sort of behavior from Chrome? Any theories as to why Chrome incorrectly added the host to HSTS?

Thanks

- D

[Chris Bentzel]

A more complete netlog would help us pinpoint the issue. Could you go to chrome://net-export and capture one? Entering this as a bug makes sense. You could also upload the netlog to Google Drive or Dropbox or equivalent and restrict access if there is private information you don't want to be seen broadly.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/2caeec0c-ac28-4d28-9778-692cab961575%40chromium.org.

[Dan L]

On Monday, June 26, 2017 at 8:52:54 AM UTC-7, Chris Bentzel wrote:
> A more complete netlog would help us pinpoint the issue. Could you go to chrome://net-export and capture one? Entering this as a bug makes sense. You could also upload the netlog to Google Drive or Dropbox or equivalent and restrict access if there is private information you don't want to be seen broadly.
>

I could but at this point I can only capture the symptom, and not the problem. The symptom is that Chrome redirects a request for http://proxysg to https://proxysg. The problem (that I can't yet replicate) is that Chrome believes proxysg is enforcing HSTS and so adds the host name to its HSTS cache.

I'm hesitant to enter it as a bug until I'm able to replicate the problem.

[Nick Harper]

I wonder if there's a bug (although I think this is unlikely) where if
a site sends the STS header thru the proxy, that it gets applied to
the proxy instead of (or in addition to) the proxied host. I think the
following would work to test if this is the case:
- Load hstspreload.org (or any other site that sends the STS header)
thru the proxy
- Check chrome://net-internals/#hsts to see if there's an entry for proxysg

> --
> You received this message because you are subscribed to the Google Groups "net-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
> To post to this group, send email to net...@chromium.org.

> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/bdaa593b-6d69-42b3-a42b-ce86bd714995%40chromium.org.

[Matt Menke]

If the proxy is transparent, we don't even know we're talking to a proxy, so there's actually nothing to set an HSTS on.  Only way we can get the domain to set HSTS on isn't the initial proxied redirect, but the request that's redirected to http://proxysg/.

On Mon, Jun 26, 2017 at 2:28 PM, Nick Harper <nha...@chromium.org> wrote:

I wonder if there's a bug (although I think this is unlikely) where if
a site sends the STS header thru the proxy, that it gets applied to
the proxy instead of (or in addition to) the proxied host. I think the
following would work to test if this is the case:
- Load hstspreload.org (or any other site that sends the STS header)
thru the proxy
- Check chrome://net-internals/#hsts to see if there's an entry for proxysg

On Mon, Jun 26, 2017 at 11:06 AM, Dan L <wrathsk...@gmail.com> wrote:
> On Monday, June 26, 2017 at 8:52:54 AM UTC-7, Chris Bentzel wrote:
>> A more complete netlog would help us pinpoint the issue. Could you go to chrome://net-export and capture one? Entering this as a bug makes sense. You could also upload the netlog to Google Drive or Dropbox or equivalent and restrict access if there is private information you don't want to be seen broadly.
>>
>
>
> I could but at this point I can only capture the symptom, and not the problem. The symptom is that Chrome redirects a request for http://proxysg to https://proxysg. The problem (that I can't yet replicate) is that Chrome believes proxysg is enforcing HSTS and so adds the host name to its HSTS cache.
>
> I'm hesitant to enter it as a bug until I'm able to replicate the problem.
>
> --
> You received this message because you are subscribed to the Google Groups "net-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+unsubscribe@chromium.org.

> To post to this group, send email to net...@chromium.org.
> To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/bdaa593b-6d69-42b3-a42b-ce86bd714995%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+unsubscribe@chromium.org.

To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CACdeXiLrvhe0dP9Rg%3DW25O-7_-LVV_93n9T5%2BV0cq%3DocuzxjXg%40mail.gmail.com.

[dly...@placer.ca.gov]

I'm actively surveying Chrome users here for this issue, and I ran across two more cases today. In one, when I queried Chrome's HSTS cache for the domain "proxysg", it returned "not found". Yet, when I deleted that domain, the problem went away.

In the second case, query domain for "proxysg" returns the following

static_sts_domain:
static_upgrade_mode: UNKNOWN
static_sts_include_subdomains:
static_sts_observed:
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: proxysg
dynamic_upgrade_mode: STRICT
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1493133654.554964
dynamic_pkp_domain:
dynamic_pkp_include_subdomains:
dynamic_pkp_observed:
dynamic_spki_hashes:

[David Benjamin]

Even if the host got into the HSTS list, I don't think we'd pay attention to the HSTS entry unless it were the origin URL being fetched. Please attach a net-internals log of this happening. Capturing the header getting installed is obviously preferable, but a capture of the entry kicking in would be helpful in seeing what kinds of requests you're making.

--
You received this message because you are subscribed to the Google Groups "net-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.

To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/d1ccc1c3-fd37-45b4-bad9-757d62e1a977%40chromium.org.

[Dan L]

On Monday, June 26, 2017 at 12:16:30 PM UTC-7, davi...@chromium.org wrote:
> Even if the host got into the HSTS list, I don't think we'd pay attention to the HSTS entry unless it were the origin URL being fetched. Please attach a net-internals log of this happening. Capturing the header getting installed is obviously preferable, but a capture of the entry kicking in would be helpful in seeing what kinds of requests you're making.
>

When a user without an existing login session to the proxy accesses any HTTP site, the proxy redirects the request to http://proxysg, by spoofing the server and issuing a 302. The browser follows that redirect and makes a direct connection to the host proxysg:80. On affected machines, it's this connection that Chrome seems to consider subject to HSTS, redirecting it to https://proxysg.

I'll do some testing with the next instance I find and try to get a net-internals log.

- D

[David Benjamin]

I see. In that case, there is at least a path for the header to get installed, be it Chrome's or the proxy's bug. (Glancing through the code, I don't see an obvious way an HSTS entry would be installed except if the URL is being fetched.)

If the proxy at https://proxysg:4443 is accidentally serving an HSTS header somehow, that would do it. For better or worse, an HSTS header on a funny port applies to the whole host and http + port 80 is indeed redirected to https + port 443. If you don't want that behavior, don't serve an HSTS header.

It's also possible Chrome is getting confused somehow, though I don't see an obvious way for that to happen, at least with the version of the code as it is? This is an up-to-date version of Chrome I assume? (Or attach a NetLog, which will include all this.)

David

[dly...@placer.ca.gov]

On Monday, June 26, 2017 at 1:00:38 PM UTC-7, davi...@chromium.org wrote:

I ran across another affected machine and was able to grab a log from net-export.

https://placercounty.box.com/s/p0n42tpuju886u02tmamhchf47opeigh

I also confirmed that when authenticating to the proxy for HTTPS access using https://proxysg:4443, there's no strict transport header delivered.

- D

[dly...@placer.ca.gov]

FWIW, at least one machine that had the problem described above, and that I had manually removed the proxy's entry from Chrome's HSTS cache, has now reverted back.

I'm still at a loss how that entry is populated into the HSTS cache when the proxy does not provide an HSTS header when users directly access it for authentication.

- D

[Matt Menke]

Looking at that log, I suspect the HSTS entry was added before the log started.  We really need to see a log that captures the transition from working to not to make any progress.

While this certainly could be an obscure Chrome bug, in more than 9 times out of 10, when people roll their own server and a browser breaks, it's a problem with the custom server.  If Chrome often added random HSTS entries, we'd presumably get a lot of bug reports about it.  Of course, it could certainly be that your server is somehow tickling a bug no one else is running into.

- D

--
You received this message because you are subscribed to the Google Groups "net-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+unsubscribe@chromium.org.

To post to this group, send email to net...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/f02cb411-0dea-4d83-ab18-49dadd42731c%40chromium.org.

[dly...@placer.ca.gov]

On Friday, July 7, 2017 at 4:35:49 PM UTC-7, Matt Menke wrote:
> Looking at that log, I suspect the HSTS entry was added before the log started. We really need to see a log that captures the transition from working to not to make any progress.

Agreed. then again, if I knew under what conditions the entry would find its way into Chrome's HSTS cache, I wouldn't be here :-)

> While this certainly could be an obscure Chrome bug, in more than 9 times out of 10, when people roll their own server and a browser breaks, it's a problem with the custom server.

The proxy is a Blue Coat Proxy ASG (now a Symantec product), so not home-rolled, but a commercial HTTP proxy with 15+ years of production in large corporate environments. I've brought this issue to Blue Coat support, but they're clueless about it.

Until I can catch the transition behavior in the wild, this will likely remain a mystery. I'm here looking for clues and hints that might lead me to a narrower set of conditions for that behavior.

Thanks

- D

[Martijn Croonen]

You can try to use the dynamic_sts_observed value from chrome://net-internals to find the URLs the user visited around that time. The dynamic_sts_observed value is a unix timestamp that is set to the current time when Chrome observes a HSTS header for the domain. Those URLs might be good candidates to try and reproduce the problem and figure out where the header is coming from.