TLS 1.2 when TLS 1.3 is enabled, Erlang 26(.1.2) and RabbitMQ 3.12(.6)

[Gert van den Berg]

I'm struggling with TLS 1.2 on RabbitMQ 3.12 on Erlang 26. (only if TLS 1.3 is enable - it works fine if TLS 1.3 is disabled)

I managed to reproduce the issue with the RabbitMQ 3.13-rc Docker image as well. (the 3.12 Docker image has Erlang 25)

To test in the container run:

apt update && apt install -y testssl.sh && testssl localhost:5671

(Also check `testssl --ssl-native localhost:5671`, which will show TLS 1.2 support, unlike the first one)

Some clients that doesn't support TLS 1.3 struggles to connect. testssl's probe for TLS 1.2 connectivity is one of those. (we had a .NET app using an unknown version of the client library on Windows Server 2016 with the problem originally)

OpenSSL works... (as does testssl --ssl-native,which is likely openssl as well)

openssl s_client -tls1_2 -connect hostname:5671 # This connects fine

I have some pcaps from ClientHellos... The one (from testssl) that gets rejected due to no suitable ciphers has all the ciphers that the working one from OpenSSL has and several more (128 ciphers instead of 28). I suspect it might be something else in the ClientHello causing issues, I haven't tracked that down yet - I don't have an easy tool to craft a ClientHello to experiment.

The error in the log when some TLS 1.2 clients tries to connect
2023-10-18 12:33:15.007908+00:00 [notice] <0.732.0> TLS server: In state hello at tls_handshake.erl:354 generated SERVER ALERT: Fatal - Insufficient Security
2023-10-18 12:33:15.007908+00:00 [notice] <0.732.0>  - no_suitable_ciphers

This feels like a bug (probably in Erlang 26...) What is the options to get this fixed?

(A workaround is to stay on Erlang 25, but that will likely stop working at some point in the future)

YAML to deploy is attached. There are 2 configmaps for rabbitmq.conf - the rabbitmq-conf12 version disables TLS 1.3 to show how TLS 1.2 then works correctly. (it is a quick edit on the deployment to point it to that configmap instead) (The rabbitmq container image's tag can be changed to 3.12 as well to see how Erlang 25 handles it)

Thanks,
Gert van den Berg

[Luke Bakken]

Hello,

I don't see any CA certificates in your configuration. Erlang has no knowledge of the system CA store by default.

You also are limiting the ciphers being used, which increases the likelihood of these sorts of issues.

Start with a "bare bones" RabbitMQ config using TLS 1.2. Don't limit ciphers, and be sure to include the full chain of CA and intermediate certs by using the cacertfile setting.

Thanks,

Luke

[Gert van den Berg]

On Wednesday, October 18, 2023 at 4:46:43 PM UTC+2 Luke Bakken wrote:

Hello,

I don't see any CA certificates in your configuration. Erlang has no knowledge of the system CA store by default.

In our normal deployment the intermediate CA is in the certfile. (it is certbot's fullchain.pem) (The test one has a self-signed cert that I can enable TLS)

We do not use client cert authentication. (It should not need to validate against CAs?) (The Ansible template generating the config does specify cacertfile and some extra option if client cert auth is used)

{%   if rabbitmq_tls_cacert_source is defined and rabbitmq_tls_cacert_source %}
# Client certificate authentication options
ssl_options.cacertfile   = /etc/rabbitmq/client_auth_ca_list.pem
ssl_options.verify     = verify_peer
ssl_options.depth = {{ rabbitmq_tls_verify_depth | default('1') }}
ssl_options.fail_if_no_peer_cert = {{ rabbitmq_tls_fail_if_no_peer_cert | default('false') }}

{%     if rabbitmq_tls_cert_login_from is defined and rabbitmq_tls_cert_login_from %}
# Configure where in the certificate the username comes from
ssl_cert_login_from = {{ rabbitmq_tls_cert_login_from }}
{%       if rabbitmq_tls_cert_login_san_type is defined and rabbitmq_tls_cert_login_san_type %}
ssl_cert_login_san_type = {{ rabbitmq_tls_cert_login_san_type }}
{%       endif %}
{%       if rabbitmq_tls_cert_login_san_index is defined and rabbitmq_tls_cert_login_san_index %}
ssl_cert_login_san_index = {{ rabbitmq_tls_cert_login_san_index | default('0') }}
{%       endif %}
{%     else %}
# We are using the DN as username, as that is the default
{%     endif %}

auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = AMQPLAIN
auth_mechanisms.3 = EXTERNAL
{%   else %}
# Client certificate auth is not configured
ssl_options.verify     = verify_none
ssl_options.fail_if_no_peer_cert = false
{%   endif %}

You also are limiting the ciphers being used, which increases the likelihood of these sorts of issues.

 

I'll try it with no cipher limiting, but the ClientHellos did in all cases include some overlap with the ciphers in the RabbitMQ config.

Start with a "bare bones" RabbitMQ config using TLS 1.2. Don't limit ciphers, and be sure to include the full chain of CA and intermediate certs by using the cacertfile setting.

TLS 1.2 works fine if TLS 1.3 is not enabled, but it fails with some clients if 1.3 is enabled. (Only on Erlang 26)

This config still results in TLS 1.2 failing from testssl: (on the 3.13-rc image on Erlang 26)

--------------------

listeners.ssl.default = 5671

ssl_options.certfile   = /etc/rabbitmq/certs/tls.crt
ssl_options.keyfile    = /etc/rabbitmq/certs/tls.key

ssl_options.verify     = verify_none
ssl_options.fail_if_no_peer_cert = false

ssl_options.versions.1 = tlsv1.2
ssl_options.versions.2 = tlsv1.3

# Logging
log.connection.level = info
log.channel.level = error

# container tricks
log.console = true

[Luke Bakken]

Hello,

It's difficult for me to assist you when I don't have your full certificates. So, if I suggest doing something (like using the cacertfile setting), I would appreciate if you give it a try.

I realize there is a cert and key in the yaml you provided, but there are no CA certs or intermediates.

If I have time I will try to reproduce this using some of my test code (https://github.com/lukebakken/erlang-tls-misc) and self-signed certs generated by the RabbitMQ team's project (https://github.com/rabbitmq/tls-gen).

What would help me out is if you could provide all of the full commands you are running to test TLS handshakes. Even better would be a script I can run. I see " openssl s_client -tls1_2 -connect hostname:5671" but you also mention "testssl".

Thanks,

Luke

[Gert van den Berg]

apt update && apt install -y testssl.sh && testssl localhost:5671

This is the test command from inside the container. (Install testssl since it is not included in the container image, run it against localhost) (and then check if it detects TLS 1.2)

(And "openssl s_client -connect localhost:5671 -tls1_2", which works)

(I suspect it is something like specific ciphers in the ClientHello or some extensions in the ClientHello that messes with it)

The difference between Erlang versions.

This has the pcap - the first two ClientHellos is openssl s_client, the later 2 is testssl while it is probing for the supported protocols (at the very start of the scan when it is generating this list) (The pcap contains hostnames, so I did not really want to send it)

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    not offered
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 not offered

(The result of that differs with the --ssl-native option, when it is using openssl for the testing instead of its own implementation)

On the pcap:

It has 4 connections - the first two is openssl s_client (TLS 1.3) (ClientHello packet 4), openssl s_client -tls1_2, (ClientHello packet 23) the second two is testssl's TLS 1.3 probe (3rd one) (ClientHello packet 43) and testssl's TLS 1.2 probe (4th / last one) (ClientHello packet 61)

(The openssl TLS 1.2 one's ServerHello will show that the chain is being sent) (packet 29) (it did not seem to make a difference, so I included a minimal cert in the test case) (I can't share one with private keys for our domains)

The pcap was with the production setup (used since RabbitMQ 3.8.11 on Erlang 24) with the ciphers as in the initial YAML.

The rejected ClientHello has ciphers that overlaps - it seems to have all supported ciphers. {The one from the service on Windows had less, but there was overlap of 6 ciphers IIRC - that cipher list is on the Slack thread somewhere) (I don't have a pcap of that ClientHello, which might have given some more clues if it is the presence of a specific cipher or the presence / absence of an extension or compression method that breaks it - there are some differences between openssl and testssl in those)

I will see if I can get something to craft a ClientHello to see if I can figure out what breaks it. (testssl's one includes 128 ciphers, which might be every possible TLS 1.2 cipher)

Gert

[Luke Bakken]

Hi Gert,

You're giving me a lot of information here, but none of it will help me figure out why you're having this issue.

I have created the following repository - https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU

https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU/blob/main/README.md

It includes self-signed certs, and some scripts to start RabbitMQ and run "openssl s_client" against it. Note that I am using a Ubuntu 22.04 environment. The testssl.sh command is not working correctly as I note in my README.

I can't reproduce what you report using Erlang 26.1.2.

What would help me out is if you could fork my repository and open a pull request with code for a client application that will demonstrate your issue.

I've spent about an hour on this and can't dedicate any more time unless you provide a means for me to reproduce this every time.

Thanks,

Luke

[Gert van den Berg]

Found what makes a difference in the ClientHello.

On Erlang 26, if TLS v1.3 is enabled, the TLS 1.2 ClientHello's signature_algorithms extension must contain sha256+rsaepss ID: 0x0804, name in Wireshark: rsa_pss_rsae_sha256.

When TLS 1.3 is disabled, the TLS 1.2 negotiation works even if no extensions are present.

Gert

Test script (requires scapy)

(The commented out extensions are the rest of what is present in the OpenSSL ClientHello. They are commented out to show that they don't make a difference)

Uncommenting any other SignatureAlgorithms with "sha256+rsaepss" commented out or commenting out the "ext=extensions" line will make it fail when talking to Rabbitmq 3.12 running on Erlang 26 with TLS 1.2 and 1.3 enabled, while it won't fail when running on Erlang 25 or if just TLS 1.2 is enabled.

--------------------------------------

#!/usr/bin/python3

import argparse

import socket

from scapy.all import *

load_layer('tls')

# Parse command-line arguments

parser = argparse.ArgumentParser(description="Send a TLS 1.2 ClientHello to a server and decode the response.")

parser.add_argument("server", help="the server to connect to")

parser.add_argument("port", type=int, help="the port to connect to")

args = parser.parse_args()

# Create a TCP socket and connect to the server

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# The cipher that gets negotiated from Windows client's ClientHello to server with TLS 1.3 disabled

min_cipher = [

        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,

        0x00ff

    ]

extensions = [

#    TLS_Ext_ServerName(servernames=ServerName(servername=args.server)),

#    TLS_Ext_SupportedPointFormat(ecpl=[0,1,2]), # openssl sends 3

#    TLS_Ext_SupportedEllipticCurves(groups=['x25519','secp256r1', 'x448', 'secp521r1', 'secp384r1']),

#    TLS_Ext_SessionTicket(),

#    TLS_Ext_EncryptThenMAC(),

#    TLS_Ext_ExtendedMasterSecret(),# extended_master_secret

    TLS_Ext_SignatureAlgorithms(sig_algs=[

#        "sha256+ecdsa",

#        "sha384+ecdsa",

#        "sha512+ecdsa",

#        "ed25519",

#        "ed448",

#        0x0809, #"sha256+rsapss",

#        0x080a, #"sha384+rsapss",

#        0x080b, #"sha512+rsapss",

        "sha256+rsaepss",

#        "sha384+rsaepss",

#        "sha512+rsaepss",

#        "sha256+rsa",

#        "sha384+rsa",

#        "sha512+rsa",

#        "sha224+ecdsa",

#        "sha224+rsa",

#        "sha224+dsa",

#        "sha256+dsa",

#        "sha384+dsa",

#        "sha512+dsa",

    ])

]

sock.connect((args.server, args.port))

hello = TLSClientHello(

    version="TLS 1.2",

    ciphers=min_cipher,

    ext=extensions

    )

# Create a TLS ClientHello packet

ch = TLS(version="TLS 1.2", msg=hello)

ch.show()

# Send the packet to the server

sock.sendall(bytes(ch))

# Receive the response from the server

response = sock.recv(4096)

sock.close()

# Parse the response as a TLS packet

tls_packet = TLS(response)

# Print the parsed packet

print(tls_packet.show())

--------------------------------------

[Luke Bakken]

Thank you, I will give this a try. This may or may not actually be a bug in Erlang's TLS implementation.

Which operating system are you running this test code on?

What Python version?

I want to be sure to test using as similar an environment as possible.

[Luke Bakken]

Tell me about this comment:

"The cipher that gets negotiated from Windows client's ClientHello "

My guess is that the issue lies in your client application, about which you haven't provided any details.

• What version of Windows does it run on?
• What programming language?
• Which AMQP client library?

Thanks,

Luke

[Luke Bakken]

Gert,

I have added your test script here:

https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU/blob/main/py/repro.py

When I start RabbitMQ using Erlang 26.1.2 (via run-rabbitmq.sh) and run your test script, I do not see any errors.

Steps to run the test script:

cd py

pipenv install

python ./repro.py localhost 5671

In your previous message, you gave very confusing information about how to reproduce the issue (comment this out, uncomment that out, etc).

Please provide EXACT code for me to run to see the issue. Fork my repository and open a pull request with the changes in your script so that when I run it, I see the issue. I'm sorry if you think I should just be able to figure this out but I simply don't have time to guess and poke at your code to get it to reproduce the issue.

I also need to know more information about the real client application that has this issue.

If I'm able to reproduce this, I will open an issue with the Erlang team, but at this point I won't because I don't have a clear set of steps to reproduce the issue.

Thanks,

Luke

[Gert van den Berg]

I created the PR.

On Thursday, October 19, 2023 at 7:32:42 PM UTC+2 Luke Bakken wrote:

Tell me about this comment:

"The cipher that gets negotiated from Windows client's ClientHello "

My guess is that the issue lies in your client application, about which you haven't provided any details.

• What version of Windows does it run on?

Windows Server 2016 

• What programming language?

C# - .NET Core 3.1 (older code)

• Which AMQP client library?

RabbitMQ.Client (unknown version - the project includes our abstraction library, depending on which version of the app is actually deployed it is 6.2.4 or 5.1.0) (the .NET Core 3.1 has me thinking 5.1.0)

Thanks,

Luke

(Some of that details was on the initial Slack thread)

https://rabbitmq.slack.com/archives/C1EDN83PA/p1697546526189799

openssl s_client works since it seems to always include those signature_algorithms (There is a -sigalgs parameter that might allow for reproducing it there as well)

testssl from their github seems to not get caught be the issue. 3.0 does.

Gert

[Luke Bakken]

Thanks Gert,

If you can, try your application using a newer (and supported) version of Windows. Since it's .NET it will use the Windows TLS implementation (not OpenSSL) and I bet there have been fixes.

I will give your code a try this coming Monday. Thanks again and have a good weekend.

[Luke Bakken]

One other thing -

If I put together a simple .NET core 3.1 app that uses the RabbitMQ.Client library, and connect using TLS on Windows Server 2016, I should see this error, yes?

Providing that I have RabbitMQ configured this way and using Erlang 26:

https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU/blob/main/rabbitmq.conf.in#L13-L14

[Gert van den Berg]

On Saturday, October 21, 2023 at 5:45:16 PM UTC+2 Luke Bakken wrote:

One other thing -

If I put together a simple .NET core 3.1 app that uses the RabbitMQ.Client library, and connect using TLS on Windows Server 2016, I should see this error, yes?

Providing that I have RabbitMQ configured this way and using Erlang 26:

https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU/blob/main/rabbitmq.conf.in#L13-L14

It should. (I can see if I can get one of the devs this side can put something together.... (or figure out enough C# to get something basic running))

Gert

[Luke Bakken]

Hey Gert,

I'm pretty sure I already have C# code to start a TCP connection with TLS.

I'll get back to this discussion within a day or two!

Thank you for all of your assistance -

Luke

[Luke Bakken]

Hello,

Well, it took a while to get everything set up, but I can reproduce this issue using Windows Server 2016 and a simple C# test client - https://github.com/lukebakken/rabbitmq-users-X9VRDGdn5HU/blob/main/cs/Program.cs

I will refine my repository so that it is very easy to get an environment running, and will open an issue in https://github.com/erlang/otp/issues

[Luke Bakken]

Alright, here is the issue: https://github.com/erlang/otp/issues/7774

The OTP team is responsive, so we should hear something within a week or two.

Thanks for all of your assistance Gert.