Intent to Deprecate and Remove: insecure TLS version fallback

[David Benjamin]

Primary eng (and PM) emails
davi...@chromium.org, awha...@chromium.org

Summary
TLS has a version negotiation mechanism to securely introduce new versions without breaking compatibility. Yet some buggy servers (usually called “version-intolerant”) implemented this wrong, so browsers were forced to add insecure fallbacks to work around this. On failed connections, we retry, advertising lower and lower maximum versions.

The SSL 3.0 fallback leg was removed last year due to POODLE (followed by removing SSL 3.0 entirely). Then in Chrome 45, we removed another leg as a trial run to removing the whole thing. We would like to finish the job and remove it completely.

This does not remove support for TLS 1.0 or TLS 1.1. Although servers are urged to upgrade to TLS 1.2 (all ciphers in prior versions have known problems), Chrome will continue to support TLS 1.0 and TLS 1.1 at this time. Only buggy servers which do not correctly implement TLS will be affected.

Motivation
The fallback bypasses TLS's downgrade protection. Attackers can force all sites (not just the buggy ones) to negotiate weaker versions, despite both client and server supporting newer, more secure protocol versions. This causes problems when attacks are found against old protocol versions, such as POODLE. FALLBACK_SCSV (RFC 7507) fixes this, but it requires both client and server support.

TLS fallbacks also mask implementation bugs. We’ve seen many examples of vendors shipping new server software with bugs that could have been noticed were it not for the fallback.

Compatibility risk
Any remaining 1.2-intolerant sites will break. After Chrome 45 removed all but the TLS 1.1 fallback, these must be 1.2-intolerant but not 1.1-intolerant. We are not aware of any such servers. (One would hope these bugs have been resolved as this is the third version bump since SSL 3.0.) Mozilla has already removed the fallback as of Firefox 37, so any 1.2-intolerant sites would not load in Firefox.

There is an additional risk that some unrelated server bug was masked by the fallback and doesn’t manifest in other browsers. Notably, we are aware of an issue affecting Chrome connections to a small fraction of Windows-based servers, though it appears much less prevalent than previously believed. (Metrics below.)

Alternative implementation suggestion for administrators
Affected sites will fail to load with a dedicated network error code, ERR_SSL_FALLBACK_BEYOND_MINIMUM_VERSION. (This page already exists from previous iterations of this.) We’ll link the error page to a Help Center article with appropriate instructions for administrators:

Server administrators should ensure their software is up-to-date and, if still unresolved, reach out to their software vendor. Affected software vendors should fix bugs in their server implementations and follow the protocol specification. The Windows issue above was addressed in KB3042058. Note that KB3042058 describes important prerequisites that must be installed prior to installing KB3042058.

For enterprise administrators, where our metrics are underrepresented, the SSLVersionFallbackMin administrative policy will be temporarily available to give more time to fix servers.

Usage information from UMA
Net.ConnectionUsedSSLVersionFallback2 measures HTTPS requests which required the fallback. Over the past week, 0.0017% of HTTPS requests required the fallback.

A few releases ago, in Chrome 46, the number was 0.015%. The difference was previously believed to be the Windows bug, but it seems to have been a different issue which is no longer a concern. (We experimentally worked around the bug, and numbers went down. Then we deprecated DHE for unrelated reasons, and then removed the experiment, but numbers stayed down. We've since observed a DHE-specific issue, also affected by the workaround, which we believe was the source.)

OWP launch tracking bug
https://crbug.com/583787

Entry on the feature dashboard
https://www.chromestatus.com/feature/5685183936200704

Requesting approval to remove too?
Yes.

As an extra precaution, the removal will be gated on field trial, so if anything unexpected occurs, we can easily roll back. We’ll also continue to monitor metrics leading up to Chrome 50 reaching stable.

[Rick Byers]

LGTM1

I especially appreciate the use of a field trial here.  If you have new uncertainty or concern, I assume you could easily start with a 1% trial and listen for feedback before ramping up?

I assume the metric indicates that some resource failed, not necessarily the main resource, right?  So this doesn't necessarily mean 0.001% of page loads will be completely broken?

Rick

[David Benjamin]

On Thu, Feb 4, 2016 at 8:01 PM Rick Byers <rby...@google.com> wrote:

LGTM1

I especially appreciate the use of a field trial here.  If you have new uncertainty or concern, I assume you could easily start with a 1% trial and listen for feedback before ramping up?

I was mostly thinking an emergency off switch, but that's certainly another option.

My gut feeling is sharding by population rather than time risks admins not reproducing the problem (or the trial is so small the admin doesn't even notice). But I've no experience to support or counter that. I can certainly try it if the numbers change or something comes up.

I assume the metric indicates that some resource failed, not necessarily the main resource, right?  So this doesn't necessarily mean 0.001% of page loads will be completely broken?

Yeah, it's out of HTTPS requests rather than page loads. (The metric lives in the net stack which doesn't know about groupings into pages.)

More specifically, it's: for each HTTPS request that ultimately succeeded (we still have the fallback today, so affected requests still go through), it records whether it succeeded normally or if it failed (on a fallback triggering-error) and then succeeded on the subsequent fallback attempt.

Oh, something I forgot to mention: ERR_CONNECTION_CLOSED is a fallback trigger, so some transient network hiccups are indistinguishable from version-intolerant servers. This means metrics have some false-positive noise. I suspect much of that noise is swallowed by how we measure deprecated cipher suites (a different kind of fallback), but it's hard to measure this. But this can only make the true number lower, and I figured 0.001% is pretty low. :-)

David

[Chris Harrelson]

LGTM2. Thank you for the very clear and thorough description.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Philip Jägenstedt]

On Fri, Feb 5, 2016 at 8:28 AM, David Benjamin <davi...@chromium.org> wrote:

On Thu, Feb 4, 2016 at 8:01 PM Rick Byers <rby...@google.com> wrote:

LGTM1

I especially appreciate the use of a field trial here.  If you have new uncertainty or concern, I assume you could easily start with a 1% trial and listen for feedback before ramping up?

I was mostly thinking an emergency off switch, but that's certainly another option.

My gut feeling is sharding by population rather than time risks admins not reproducing the problem (or the trial is so small the admin doesn't even notice). But I've no experience to support or counter that. I can certainly try it if the numbers change or something comes up.

I assume the metric indicates that some resource failed, not necessarily the main resource, right?  So this doesn't necessarily mean 0.001% of page loads will be completely broken?

Yeah, it's out of HTTPS requests rather than page loads. (The metric lives in the net stack which doesn't know about groupings into pages.)

More specifically, it's: for each HTTPS request that ultimately succeeded (we still have the fallback today, so affected requests still go through), it records whether it succeeded normally or if it failed (on a fallback triggering-error) and then succeeded on the subsequent fallback attempt.

Oh, something I forgot to mention: ERR_CONNECTION_CLOSED is a fallback trigger, so some transient network hiccups are indistinguishable from version-intolerant servers. This means metrics have some false-positive noise. I suspect much of that noise is swallowed by how we measure deprecated cipher suites (a different kind of fallback), but it's hard to measure this. But this can only make the true number lower, and I figured 0.001% is pretty low. :-)

The way the metric is implemented, it sounds like it doesn't distinguish at all between the top-level resource and sub-resources? If up to 0.001% of page loads are fatally wounded either by the top-level fetch or some necessary sub-resource, that could be a very large number of affected users and servers.

What will this look like for non-Chrome browsers? Will flag for insecure TLS version fallback be off by default until the field trial has concluded and the flag is cleaned out of the source code?

At any rate, LGTM3, as this seems like an important fix, and with the ability to roll back the flag even after reaching stable really limits the potential for mayhem.

Philip

[Rick Byers]

On Thu, Feb 4, 2016 at 8:28 PM, David Benjamin <davi...@chromium.org> wrote:

On Thu, Feb 4, 2016 at 8:01 PM Rick Byers <rby...@google.com> wrote:

LGTM1

I especially appreciate the use of a field trial here.  If you have new uncertainty or concern, I assume you could easily start with a 1% trial and listen for feedback before ramping up?

I was mostly thinking an emergency off switch, but that's certainly another option.

My gut feeling is sharding by population rather than time risks admins not reproducing the problem (or the trial is so small the admin doesn't even notice). But I've no experience to support or counter that. I can certainly try it if the numbers change or something comes up.

Yeah that's a really good point - when it's important to fix a problem, ripping off the band-aid can be better then introducing non-determinism.

[David Benjamin]

On Fri, Feb 5, 2016 at 8:42 AM Philip Jägenstedt <phi...@opera.com> wrote:

The way the metric is implemented, it sounds like it doesn't distinguish at all between the top-level resource and sub-resources? If up to 0.001% of page loads are fatally wounded either by the top-level fetch or some necessary sub-resource, that could be a very large number of affected users and servers.

That's true. Though this is all a property of the host, not the full URL (we don't even send the path over the wire until the handshake happens), so it's not every random subresource in a page, just the distinct hosts. I don't recall a problem from decisions based on net-stack vs. page-level metrics before (even when we look at connection-level rather than request-level metrics), and I think the current numbers have quite a lot of wiggle room.

The alternative is a huge amount of effort plumbing all this information up the stack. The low-level parts of the net stack and Blink are quite far apart.

 

What will this look like for non-Chrome browsers? Will flag for insecure TLS version fallback be off by default until the field trial has concluded and the flag is cleaned out of the source code?

My intention was for the flag to be a temporary emergency off switch rather than a custom roll-out (the usual release rollout seems fine), so I'm expecting to have the fallback disabled by default, yeah.

[Philip Jägenstedt]

On Fri, Feb 5, 2016 at 10:59 PM, David Benjamin <davi...@chromium.org> wrote:

On Fri, Feb 5, 2016 at 8:42 AM Philip Jägenstedt <phi...@opera.com> wrote:

The way the metric is implemented, it sounds like it doesn't distinguish at all between the top-level resource and sub-resources? If up to 0.001% of page loads are fatally wounded either by the top-level fetch or some necessary sub-resource, that could be a very large number of affected users and servers.

That's true. Though this is all a property of the host, not the full URL (we don't even send the path over the wire until the handshake happens), so it's not every random subresource in a page, just the distinct hosts. I don't recall a problem from decisions based on net-stack vs. page-level metrics before (even when we look at connection-level rather than request-level metrics), and I think the current numbers have quite a lot of wiggle room.

Yeah, the numbers you are enough to makes a guess about the risk, just trying to understand what the worst case is based on the 0.001% figure.

 

The alternative is a huge amount of effort plumbing all this information up the stack. The low-level parts of the net stack and Blink are quite far apart.

 

What will this look like for non-Chrome browsers? Will flag for insecure TLS version fallback be off by default until the field trial has concluded and the flag is cleaned out of the source code?

My intention was for the flag to be a temporary emergency off switch rather than a custom roll-out (the usual release rollout seems fine), so I'm expecting to have the fallback disabled by default, yeah.

Oh... I don't know much about Chrome field trials, is it such that you can flip the flag without making a new release in case disaster strikes? For any Chromium-based browser without an equivalent mechanism, it would be stuck in the storm until an emergency release can be made, it sounds like.

Would it be possible to leave the fallback enabled by default and change it once the field trial has proven it web compatible on stable? I realize that this amount to Opera taking not sharing in the risk, but the alternative seems to be taking a much higher risk.

Philip

[David Benjamin]

On Fri, Feb 5, 2016 at 11:41 AM Philip Jägenstedt <phi...@opera.com> wrote:

Oh... I don't know much about Chrome field trials, is it such that you can flip the flag without making a new release in case disaster strikes? For any Chromium-based browser without an equivalent mechanism, it would be stuck in the storm until an emergency release can be made, it sounds like.

Would it be possible to leave the fallback enabled by default and change it once the field trial has proven it web compatible on stable? I realize that this amount to Opera taking not sharing in the risk, but the alternative seems to be taking a much higher risk.

I'd rather not have to fuss with pushing a field trial forward and such for what's supposed to be a low-risk default. The other time I did this was for RC4 because that ended up kind of rushed. We haven't had to flip it back. SSL 3.0 had the same arrangement. We didn't have to use it. I'm adding it here just because it's easy when I'm adding the admin policy anyway, not because we've found this useful in the past.

What parts of the code does Opera share or not share? The field trial's going to be in a pretty high-level spot (//chrome or //components). If you do ship that code, surely you all can just make base::FieldTrialList (or base::FeatureList I guess is the new one?) return the other thing for this knob, or patch out the default? Though I would appreciate it if Opera shared in this.

David

[Philip Jägenstedt]

OK, so I didn't read everything about the field trial, you said "I was mostly thinking an emergency off switch" earlier.

On the products I'm familiar with, Opera doesn't use //chrome but rather uses the content API.

I think perhaps it would be best if you proceed with the plan as you had intended, and if you point me to the relevant CLs off-list I can check what Opera's products can/should do about this. That the same setup has already been used for RC4 and SSL 3.0 is promising.

Philip

[David Benjamin]

Sounds good. And I'm certainly happy to let you and this thread know if I end up using the switch.

David 

[CarlosVa...@hotmail.com]

[rg.c...@gmail.com]

Hey guys, my bests for everyone:

Never heard about this "Juniper" thing. Now I go to my site (www.cedip.cl) and this is what returns:

Juniper Networks Firewall has blocked the URL: 

10.107.121.80(52371)->50.87.248.174(80) www.cedip.cl 

CATEGORY: Enhanced_Search_Engines_and_Portals REASON: BY_SITE_REPUTATION

With chrome, safari and mozilla is the same (MacBookPro) 

It does not happen with other computers at my clinic, using the same network.  The clinic does not use Juniper.  It´s simply irritating and extremely suspicious. 

Rick

[PhistucK]

This is not a technical support group. Seek help elsewhere.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "net-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to net-dev+u...@chromium.org.
To post to this group, send email to net...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/11107fb3-d3dc-45fa-9635-35a048d2b380%40chromium.org.

[Adrienne Porter Felt]

You can find the Chrome help forum here: https://productforums.google.com/forum/#!forum/chrome

[PhistucK]

Yes, but it happens with other browsers, so not Chrome related.

☆PhistucK

[Ricardo Gómez]

​Dear all:

Would you kindly direct me to an url useful with that purpose? I am preparing a dossier to be given to the clinic network for them to act upon fair practices. ​It is clear that Juniper has crossed privacy limits, wrongly assessed a clean site and members do not want to talk about it. 

With warm personal regards,

2016-07-19 16:03 GMT-04:00 PhistucK <phis...@gmail.com>:

This is not a technical support group. Seek help elsewhere.

☆Phist

--

Ricardo Gómez

Médico Jefe de Obstetricia, Hospital Clínico de La Florida

Profesor Asociado Adjunto, P. Universidad Católica de Chile

Director CEDIP-HLF-UC

Centro de Diagnóstico, Investigaciones Perinatales

e Innovación Docente

Hospital Clínico de La Florida - P.Universidad Católica de Chile

Jefe de la Unidad de Medicina Materno-Fetal y

Ultrasonido de Clínica Santa María

Santiago, Chile

Cedip Internet, el punto de partida de la ciudad digital del Cedip-HLF-UC para Latinoamérica

[Matt Menke]

This isn't a Chrome issue, since it happens with other browsers.  This isn't a mailing list for end user support.  So asking for help with an end user issue that isn't even with Chrome is completely out of scope, in multiple dimensions.  No one here has anything to do with Juniper, it's not part of Chrome, we don't ship it with Chrome, etc.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/net-dev/CAOemBWgYR8J0qo9EMiSvPEFoHLxOD08qnL1bZ4JCrZphqFSZzQ%40mail.gmail.com.

[leehye...@gmail.com]

2016년 2월 4일 목요일 오전 6시 42분 59초 UTC+9, David Benjamin 님의 말:

[r.keeree...@gmail.com]

เมื่อ วันพฤหัสบดีที่ 4 กุมภาพันธ์ ค.ศ. 2016 4 นาฬิกา 42 นาที 59 วินาที UTC+7, David Benjamin เขียนว่า:

[lam1...@gmail.com]

Vào 04:42:59 UTC+7 Thứ Năm, ngày 04 tháng 2 năm 2016, David Benjamin đã viết:

[dieprufi]

Vào 04:42:59 UTC+7 Thứ Năm, ngày 04 tháng 2 năm 2016, David Benjamin đã viết:

Primary eng (and PM) emails

[randle...@gmail.com]

[myas1...@gmail.com]

[lequa...@gmail.com]

Vào 04:42:59 UTC+7 Thứ Năm, ngày 04 tháng 2 năm 2016, David Benjamin đã viết:

Primary eng (and PM) emails

[cunb...@gmail.com]

Vào 04:42:59 UTC+7 Thứ Năm, ngày 04 tháng 2 năm 2016, David Benjamin đã viết:

Primary eng (and PM) emails

[cunb...@gmail.com]

a

[eusdebia...@gmail.com]

[1300...@patterson.k12.ca.us]

On Wednesday, February 3, 2016 at 1:42:59 PM UTC-8, David Benjamin wrote:

[emmanuell...@gmail.com]

[sadikh...@gmail.com]

[djamela...@gmail.com]

[jordino...@gmail.com]

[kulluranj...@gmail.com]

[rachida...@gmail.com]

بتاريخ الأربعاء، 3 فبراير، 2016 10:42:59 م UTC+1، كتب David Benjamin:

[as99...@gmail.com]

[kathy...@gmail.com]

[Habtamu Negese]

I am repeatedly encountering connection problems. so what you recommend me to fix it?