Connectivity breakage from removal of TLS RSA KEX from default encryption suite

[Creaky]

Hi,

Please forgive me if this is not the correct place to make this posting and let me know the correct place instead.

Want to discuss the impacts from the implementation of the proposal crypto/tls: disable RSA key exchange cipher suites by default #63413 and possible ways forward beyond go change some code (whilst it may fix the issue in the short term, has certain drawbacks, takes time, and is impractical in some cases).

This proposal was introduced in Go Version 1.22 for crypto/tls and brought a significant behavioural change impacting client server connectivity with very little announcement.

People are seeing TLS handshake failures causing complete failure of connectivity and not knowing where the issue is. As tooling starts to take on the 1.22 / 1.23 Golang changes, the impacts from this change are now bubbling to the surface.

I can see why the motivation behind the change is from a good place. There are weaknesses in using TLS with RSA encrypted key exchange. However, the change as implemented is producing poor outcomes.

From a high level perspective, it is far better to protect communication with possibly vulnerable encryption than being completely in the clear. However this breakage is encouraging some people to revert to in clear connectivity for lacking (unknowing) a better option.

Reading through the original proposal it appears considerations of change impact narrowly focused on Internet connected clients and servers and did not consider the various audience groups impacted by the proposed (and now accepted) change.

This change is now causing one of the more costly failure types, that being in the field failures.

What can be done now this proposal is implemented?

Can the proposal and its implementation be reversed? Or removed and reintroduced with better communication?

There are several areas of concern and possible remediation needed.

1. Scope of impact.

The proposal itself mentions insufficient metrics to understand current usage of TLS RSA KEX in the wild and discussion was limited to Internet clients and servers.

Such usage in fact includes:

• Embedded web servers
• Server out of band remote management (iLOMs etc)
• Older web acceleration / proxy servers
• Other protocol proxy servers (LDAP, IMAP etc)
• Golang monitoring systems monitoring older type of servers

Most of the above usage will not appear in Internet cipher suite usage statistics.

2. Documentation.

There is insufficient documentation around this high impact change.

Existing documentation is limited to:

• A small mention in release notes
• The proposal itself
• A couple of support tickets

There is no documentation or workarounds written up or referenced to support non programmers impacted by this change. This is a behavioural change in Golang, so relying on vendor / OSS documentation to percolate through is insufficient. Often such projects are only discovering issues at the same time as end users. The significance of the one line release note statement is easy to miss.

3. Impact of change.

The change completely breaks client server connectivity where TLS RSA KEX is the only supported cipher suite. The high impact coupled with lack of documentation, especially aimed at non programmers (think System/Network/DevOps administrators), is causing undesirable behaviours and poor security posture as the people balance the breakage against other competing goals. Some outcomes observed include:

• Revert communications to clear text (really not a desirable outcome but it has happened)
• Bypass the broken system
• Revert to older implementation and mark system / application as high risk for upgrade. (Effectively blocking all changes)

This is not a good security outcome. Even reasonable encryption is better than no encryption and lots of encryption implementations do not guarantee PFS which was one of the two major motivations behind this proposal change.

4. Ongoing support of legacy cipher suites.

There needs to be some guarantees made for:

• Keeping GODEBUG=tlsrsakex=1 for a certain time period to allow vendors, implementations and impacted communities to move forward. I would expect several years, like 5 or more, may be in order.
  
• Keeping older and deprecated cipher suites available for legacy applications accessibility.
  

As it stands, I am not aware of current policies where something hidden behind GODEBUG is maintained for any set length of time. There is a risk here of the tlsrsakex=1 workaround breaking in future (even tomorrow) and requiring yet further interventions or removal of Golang applications.

The systems impacted are often older, lower powered, potentially out of support and remain needing access at the highest security standard that existed at their time of implementation. Looking to vendor's / OSS for updates to said systems in such ecosystems is a very long process.

5. Improvements to the Proposal methodology.

This is an interesting case, as reading through Golang's documentation around proposals has lots of info about the process of proposal being accepted and implemented, but very little to nothing about post proposal acceptance and implementation ongoing review. 

The procedure basically stops at the it is implemented, released and done stage. Whereas the lifecycle of the proposal is in fact only just starting as the proposal's implementation is now introduced into the wild.

Maybe these issues could be a good candidate case for sparking discussions on improving proposal considerations and post implementation review.

Regards,

Creaky

[Ian Lance Taylor]

On Mon, Aug 26, 2024 at 8:31 AM Creaky <whatex...@gmail.com> wrote:
>
> Want to discuss the impacts from the implementation of the proposal crypto/tls: disable RSA key exchange cipher suites by default #63413 and possible ways forward beyond go change some code (whilst it may fix the issue in the short term, has certain drawbacks, takes time, and is impractical in some cases).
>
> This proposal was introduced in Go Version 1.22 for crypto/tls and brought a significant behavioural change impacting client server connectivity with very little announcement.
>
> People are seeing TLS handshake failures causing complete failure of connectivity and not knowing where the issue is. As tooling starts to take on the 1.22 / 1.23 Golang changes, the impacts from this change are now bubbling to the surface.
>
> I can see why the motivation behind the change is from a good place. There are weaknesses in using TLS with RSA encrypted key exchange. However, the change as implemented is producing poor outcomes.
>
> From a high level perspective, it is far better to protect communication with possibly vulnerable encryption than being completely in the clear. However this breakage is encouraging some people to revert to in clear connectivity for lacking (unknowing) a better option.
>
> Reading through the original proposal it appears considerations of change impact narrowly focused on Internet connected clients and servers and did not consider the various audience groups impacted by the proposed (and now accepted) change.
>
> This change is now causing one of the more costly failure types, that being in the field failures.
>
> What can be done now this proposal is implemented?
>
> Can the proposal and its implementation be reversed? Or removed and reintroduced with better communication?

Thanks for the detailed note.

My takeaway here is that the Go team needs to provide better
communication about the consequences that people will see from this
kind of change, and how to better communicate those consequences and
how to work around them. That is, I believe that the change is a good
one for the overall ecosystem. But people can encounter this change
and not know what caused it and not know the relatively simple
workarounds that are available.

As you know, those workarounds are described at
https://go.dev/doc/godebug. People working with an executable program
can set the GODEBUG variable in the environment. People developing a
program from source can do that, and can also add a go:debug directive
to the main package and/or a godebug setting in a go.mod or go.work
file. There are no plans to remove the tlsrsakex setting.

Still you're absolutely right that the average user will see some sort
of failure to communicate, and will have no clear way to translate
that into what they need to do to change. So the question is: how can
we communicate that better? What are the right channels? Would a Go
blog entry be sufficient? Also, what kinds of errors will users see
in practice, and how can we get those errors to direct them to the
GODEBUG setting?

Thanks.

Ian

[robert engels]

Hi,

Isn’t the version release notes sufficient? Maybe add a "breaking changes" section? https://tip.golang.org/doc/go1.23

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWTRwuFjZh_FEEAnJATS_io1VxP5c%2BS4Dv_oeshX-%2B0qA%40mail.gmail.com.

[Ian Lance Taylor]

On Mon, Aug 26, 2024 at 2:28 PM robert engels <ren...@ix.netcom.com> wrote:
>
> Isn’t the version release notes shttps://pkg.go.dev/net/http#Request.Hostufficient? Maybe add a "breaking changes" section? https://tip.golang.org/doc/go1.23

This change was indeed mentioned in the release notes
(https://go.dev/doc/go1.22#minor_library_changes, the crypto/tls
entry). But there can be a pretty big gap between the person who
builds a program with a new version of Go and the user who encounters
a problem running that program. Also that release note is, frankly,
pretty cryptic as to the consequences of the change. We want to make
it as easy as possible for the user of the program to close that gap.

Ian

[robert engels]

I don’t think being concerned about the user is correct - I suspect that many (most?) of the users have no idea the program is even written in Go…

This is a builders responsibility imo to either fail at start-up with a user understandable error, add mitigating code, or don’t build and release using the latest release.

[Ian Lance Taylor]

On Mon, Aug 26, 2024 at 2:59 PM robert engels <ren...@ix.netcom.com> wrote:
>
> I don’t think being concerned about the user is correct - I suspect that many (most?) of the users have no idea the program is even written in Go…
>
> This is a builders responsibility imo to either fail at start-up with a user understandable error, add mitigating code, or don’t build and release using the latest release.

I think this case is more complex, because this involves a network
communication with two different sides. It is possible for one or the
other side to be rebuilt when the other is not.

[Robert Engels]

Hmm. Aren’t the endpoints supposed to negotiate the available cryptographic methods?

So wouldn’t this affect non Go endpoints as well - which puts the burden back on the side trying to use the latest Go version which is removing some of the methods?

Making it “if you upgrade to this version of Go you will no longer accept any clients expecting to use TLS” - unless you do X?

> On Aug 26, 2024, at 5:22 PM, Ian Lance Taylor <ia...@golang.org> wrote:

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcV_DC%3DQC_pihmac_oK0j0BjS4Lq%3DuTcLmzh34gyHLPJmg%40mail.gmail.com.

[Ian Lance Taylor]

On Mon, Aug 26, 2024 at 10:16 PM Robert Engels <ren...@ix.netcom.com> wrote:
>
> Hmm. Aren’t the endpoints supposed to negotiate the available cryptographic methods?
>
> So wouldn’t this affect non Go endpoints as well - which puts the burden back on the side trying to use the latest Go version which is removing some of the methods?
>
> Making it “if you upgrade to this version of Go you will no longer accept any clients expecting to use TLS” - unless you do X?

Yes, that is my understanding.

Ian

[Creaky]

On Tuesday, August 27, 2024 at 8:00:27 AM UTC+10 robert engels wrote:

I don’t think being concerned about the user is correct - I suspect that many (most?) of the users have no idea the program is even written in Go…

This is a builders responsibility imo to either fail at start-up with a user understandable error, add mitigating code, or don’t build and release using the latest release.

> On Aug 26, 2024, at 4:45 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>
> On Mon, Aug 26, 2024 at 2:28 PM robert engels <ren...@ix.netcom.com> wrote:
>>
>> Isn’t the version release notes shttps://pkg.go.dev/net/http#Request.Hostufficient? Maybe add a "breaking changes" section? https://tip.golang.org/doc/go1.23
>

The Go ecosystem has created the expectation that a compiled program will run correctly. Especially that of a previously compiled program will run under a new compiler version with the same behaviour.

Breaking changes that change the contract of behaviour continuity needs special mentioning and highlighting. Changing defaults is one such change.

A builder may not be cognisant of the significance or impact of a release note entry. Especially when the breaking behaviour is not noted or requires extensive understanding of a specialist technical area.

 

The users impacted (or I prefer audience set) can be broad. From programmers, deployers, administrators to end users. A good portion of the audience set will have the capability to know it is a Go program at fault.

 

The key is discoverability of the issue's cause and fix. Good documentation and clear communications aids in this.

For breaking changes, the audience needs the ability to find a solution and communication needs to support easy discovery of said solution.

The current single line Release note entry is insufficient for meeting the need of easy discovery.

Practices I have seen to address discoverability includes identifying breaking changes separately, often in its own document. Usually with a link to an accompanying technical note, bulletin or blog. 

The accompanying document then describes the change in detail including:

• Summary of what the change is.
• Reason for change.
• Impact of change, what breaks and how to identify it.
• Various solutions or fixes to restore behaviour. With source code or just with executable.
• Commitment to support or provide continuity for said solutions or fixes.

Increasing the easy of discovery helps everyone in the ecosystem.

Regards,

Creaky

[Creaky]

Endpoints negotiate common cipher suites. Change removed cipher suites compatible with clients or servers supporting only a limited set of ciphers.

Resulting error is a rather generic TLS handshake error message. Leaves the recipient with little direction to fault find on and makes finding unless you do X difficult.

Just to clarify, it is not just Go servers, Go when acting as a client is similarly impacted.

Creaky

 

[robert engels]

The statement "Especially that of a previously compiled program will run under a new compiler version with the same behaviour” is not true with many platforms. Even with Java, with a great backwards guarantee, this is not the case (i.e. running on a later JVM) - although this is a somewhat recent behavioral change.

I said to add a “breaking changes” to the release notes. If you are upgrading your compiler/stdlib version, and rebuilding the software, it would be the builders responsibility to check that.

Even if the end user knew it was a Go program, they probably can’t build it, and/or would not know how to add runtime options to work around it.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e728d646-3baa-4d26-a396-6cb741b864b5n%40googlegroups.com.

[Roland Shoemaker]

I agree that in this case the release note we provided was likely not informative enough. We try to walk the line of providing useful yet concise notes about changes, but in this particular case we did not provide enough detail specifically about how this change may cause breakage.

In general though, I still believe this change was the correct choice. Defaults in crypto/tls are one of the main ways we can provide security benefit for users, and when weighing this change we looked at what would be best for the vast majority of consumers of the library. Based on the research we did, we expected that disabling these ciphers by default would cause breakage, but only for a small number of users, which based on experience reports we've seen so far is still true.

It should also be noted that we didn't completely disable these ciphersuites, we only removed them from the default set, explicitly configuring them in the tls.Config enables them. For users who need to support a large array of endpoints, defaults are likely to not be appropriate in general, and the supported ciphersuites should be manually configured. Looking through the issues referenced from the original proposal, it looks like most downstream users who saw breakage here were able to manually configure support for these older ciphersuites, fixing the issue.

That said, the error message returned here is not particularly useful, and we should probably update it to point the user to which ciphersuites were offered, and perhaps in this particular case note that using GODEBUG=tlsrsakex=1 would fix the issue. This should be appropriate for both developers, who can manually configure the needed suites, and users who can set the GODEBUG until an upstream fix is available.

Roland

[Roland Shoemaker]

I should also note that for programs built using the go.mod "go" directive, with a version set to a pre-1.22 value, due to GODEBUG semantics the default behavior will not have changed, since it will be automatically set to the value that enables the old behavior.