3.8.10: TLS connections dropping

[Gert van den Berg]

Hi,

Does anyone have 3.8.10 working with TLS?

I had one server that was installed on it yesterday. It seems like it fails with a connection reset before sending the certificate (on AMQPS and the management interface)

The relevant server were downgraded to 3.8.9 which solved the problem.... (We do not currently use client certificate authentication, if that is relevant. The certificates were Let's Encrypt ones (and were not changed on the downgrade))

The only log entries that I have (I did not capture the s_client logs) at the time of the problematic connections:

2021-01-20 15:47:19.717 [debug] <0.1216.0> Supervisor {<0.1216.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.1217.0>

2021-01-20 15:47:19.717 [debug] <0.1216.0> Supervisor {<0.1216.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.1217.0>, {acceptor,{0,0,0,0,0,0,0,0},5671}) at pid <0.1218.0>

2021-01-20 15:47:21.505 [debug] <0.1220.0> Supervisor {<0.1220.0>,rabbit_connection_sup} started rabbit_connection_helper_sup:start_link() at pid <0.1221.0>

2021-01-20 15:47:21.505 [debug] <0.1220.0> Supervisor {<0.1220.0>,rabbit_connection_sup} started rabbit_reader:start_link(<0.1221.0>, {acceptor,{0,0,0,0,0,0,0,0},5672}) at pid <0.1222.0>

Ubuntu package, from the bintray repo, on Ubuntu 18.04 with Erlang 23.x, also from bintray.

Start-up output (for versions)

 Starting RabbitMQ 3.8.10 on Erlang 23.2.2

 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.

 Licensed under the MPL 2.0. Website: https://rabbitmq.com

2021-01-19 17:07:02.356 [info] <0.271.0>

 node           : rabbit@greenqueue-3

 home dir       : /var/lib/rabbitmq

 config file(s) : (none)

 cookie hash    : pdC5FR9bud00GfiyQ1YC0Q==

 log(s)         : /var/log/rabbitmq/rab...@greenqueue-3.log

                : /var/log/rabbitmq/rabbit@greenqueue-3_upgrade.log

 database dir   : /var/lib/rabbitmq/mnesia/rabbit@greenqueue-3

2021-01-19 17:07:34.446 [warning] <0.271.0> Erlang VM is running with 1 I/O threads, file I/O performance may worsen

2021-01-19 17:07:34.584 [info] <0.271.0> Running boot step pre_boot defined by app rabbit

2021-01-19 17:07:34.584 [info] <0.271.0> Running boot step rabbit_core_metrics defined by app rabbit

2021-01-19 17:07:34.585 [info] <0.271.0> Running boot step rabbit_alarm defined by app rabbit

2021-01-19 17:07:34.588 [info] <0.450.0> Memory high watermark set to 3084 MiB (3234027929 bytes) of 7710 MiB (8085069824 bytes) total

2021-01-19 17:07:34.592 [info] <0.452.0> Enabling free disk space monitoring

2021-01-19 17:07:34.592 [info] <0.452.0> Disk free limit set to 50MB

2021-01-19 17:07:34.595 [info] <0.271.0> Running boot step code_server_cache defined by app rabbit

2021-01-19 17:07:34.595 [info] <0.271.0> Running boot step file_handle_cache defined by app rabbit

2021-01-19 17:07:34.596 [info] <0.455.0> Limiting to approx 299903 file handles (269910 sockets)

2021-01-19 17:07:34.596 [info] <0.456.0> FHC read buffering:  OFF

2021-01-19 17:07:34.596 [info] <0.456.0> FHC write buffering: ON

2021-01-19 17:07:34.596 [info] <0.271.0> Running boot step worker_pool defined by app rabbit

2021-01-19 17:07:34.596 [info] <0.384.0> Will use 2 processes for default worker pool

2021-01-19 17:07:34.596 [info] <0.384.0> Starting worker pool 'worker_pool' with 2 processes in it

2021-01-19 17:07:34.597 [info] <0.271.0> Running boot step database defined by app rabbit

2021-01-19 17:07:34.598 [info] <0.271.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2021-01-19 17:07:34.600 [info] <0.271.0> Successfully synced tables from a peer

2021-01-19 17:07:34.600 [info] <0.271.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2021-01-19 17:07:34.600 [info] <0.271.0> Successfully synced tables from a peer

2021-01-19 17:07:34.624 [info] <0.271.0> Waiting for Mnesia tables for 30000 ms, 9 retries left

2021-01-19 17:07:34.625 [info] <0.271.0> Successfully synced tables from a peer

2021-01-19 17:07:34.625 [info] <0.271.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.

Config

----

listeners.ssl.default = 5671

ssl_options.cacertfile = /etc/letsencrypt/live/hostname.removed/chain.pem

ssl_options.certfile   = /etc/letsencrypt/live/hostname.removed/cert.pem

ssl_options.keyfile    = /etc/letsencrypt/live/hostname.removed/privkey.pem

#ssl_options.password   = MySecretPass

ssl_options.verify     = verify_peer

ssl_options.fail_if_no_peer_cert = false

ssl_options.versions.1 = tlsv1.2

ssl_options.versions.2 = tlsv1.3

# Security options

ssl_options.honor_cipher_order   = true

ssl_options.honor_ecc_order      = true

ssl_options.client_renegotiation = false

ssl_options.secure_renegotiate   = true

# Ciphers with forward secrecy

ssl_options.ciphers.1   = ECDHE-ECDSA-AES256-GCM-SHA384

ssl_options.ciphers.2   = ECDHE-RSA-AES256-GCM-SHA384

ssl_options.ciphers.3   = DHE-RSA-AES256-GCM-SHA384

ssl_options.ciphers.4   = ECDHE-ECDSA-AES128-GCM-SHA256

ssl_options.ciphers.5   = ECDHE-RSA-AES128-GCM-SHA256

ssl_options.ciphers.6   = DHE-RSA-AES128-GCM-SHA256

ssl_options.ciphers.7   = ECDH-ECDSA-AES256-GCM-SHA384

ssl_options.ciphers.8   = ECDH-RSA-AES256-GCM-SHA384

ssl_options.ciphers.9   = ECDH-ECDSA-AES128-GCM-SHA256

ssl_options.ciphers.10  = ECDH-RSA-AES128-GCM-SHA256

ssl_options.ciphers.11  = AES256-GCM-SHA384

ssl_options.ciphers.12  = AES128-GCM-SHA256

handshake_timeout = 30000

# Logging

log.connection.level = info

log.channel.level = error

# Allow remote guest connections

loopback_users = none

# Memory threshold https://www.rabbitmq.com/memory.html

# vm_memory_high_watermark.absolute = 1024MB

# Relative value seems safer

# Default of 40% of RAM is normally fine

# vm_memory_high_watermark.relative = 0.4

# Increase max message size

max_message_size = 268435456

# Per-connection channel limits

channel_max = 10

# Management plugin

# TODO: New format https://www.rabbitmq.com/management.html#configuration

management.ssl.port = 15672

management.ssl.cacertfile = /etc/letsencrypt/live/hostname.removed/chain.pem

management.ssl.certfile = /etc/letsencrypt/live/hostname.removed/cert.pem

management.ssl.keyfile = /etc/letsencrypt/live/hostname.removed/privkey.pem

management.ssl.versions.1 = tlsv1.2

management.ssl.versions.2 = tlsv1.3

management.ssl.honor_cipher_order   = true

management.ssl.honor_ecc_order      = true

management.ssl.client_renegotiation = false

management.ssl.secure_renegotiate   = true

management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384

management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384

management.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384

management.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384

management.ssl.ciphers.5 = DHE-RSA-AES256-GCM-SHA384

# Promtheus plugin

#prometheus.return_per_object_metrics = true  # Uncomment for per-queue, channel and connection stats

prometheus.tcp.port = 15692

prometheus.tcp.inactivity_timeout = 120000

prometheus.tcp.idle_timeout = 120000

# MQTT

# Note: This uses the same ssl_options as AMQPS

mqtt.listeners.ssl.default = 8883

mqtt.listeners.tcp.default = 1883

mqtt.subscription_ttl = 600000

# Example has mqt.topic

mqtt.exchange         = topic

# Metrics collection interval:

collect_statistics_interval = 10000

---
Thanks,

Gert

[Gert van den Berg]

I did verify that the certificate, key and CA is readable as well, since that might be the most likely culprit (I add group read permissions to the certs in a hook script, and added the rabbitmq user to that group) (and nothing changed on the downgrade)

(I did have to reset rabbitmq on the downgrade (it was a new install anyway, so it is not a huge issue), if that is relevant)

[Luke Bakken]

Hi Gert,

Thanks for all of the information. Just to confirm, you're using the same version of Erlang with both versions of RabbitMQ?

[Gert van den Berg]

On Thursday, January 21, 2021 at 5:50:11 PM UTC+2 Luke Bakken wrote:

Hi Gert,

Thanks for all of the information. Just to confirm, you're using the same version of Erlang with both versions of RabbitMQ?

Correct. I just downgraded RabbitMQ with APT, and cleared /var/lib/rabbitmq when the older version did not want to start (I initially went to 3.8.8, which I know works on some other servers. (I then blacklisted 3.8.10 in apt and upgraded to 3.8.9, which also works)

Gert

[Gert van den Berg]

https://groups.google.com/g/rabbitmq-users/c/erazwJE2VS4

The symptoms there looks very similar... (Connection reset before cert exchange)

A rabbit that updated to 3.8.11 shows the same connection reset on the management port... (this one is behind a VPN that I don't have access to get the logs. The config was generated by the same Ansible script as the one posted here) (Netstat showed some connections on the AMQPS port, so I'm not completely sure whether that is affected as well...)

Vagrant attached. Not updating everything - that is a bit slow on my internet (working from home issues) (It will install the latest version, which does not matter since 3.8.11 has the same issue.

In VM, do

openssl s_client -connect localhost:5671 # Note no certificate is obtained.

Do systemctl stop rabbitmq-server

rm -r /var/lib/rabbitmq/* # Something there from the new version keeps old version from starting. It does not seem needed if it was upgraded instead of installed on new one...
apt install rabbitmq-server=3.8.8-1 

openssl s_client -connect localhost:5671 # see that cert is sent

It can be tweaked for other versions by modifying the apt install line, e.g changing rabbitmq-server to rabbitmq-server=3.8.8-1

A test cert is included, in case something in there affects it...

Gert

[Luke Bakken]

Hi Gert,

I will find time to reproduce this week. Thanks.

[Luke Bakken]

Hi Gert,

Can you confirm that this issue only happens after an upgrade? Do you see the issue with a 100% fresh installation?

I'm about to test RabbitMQ 3.8.11 using certs generated by tls-gen as well as your certs.

On Tuesday, January 26, 2021 at 9:27:29 AM UTC-8 ger...@gmail.com wrote:

[Luke Bakken]

Hi Gert,

I was able to reproduce the connection error you report using the set of certificates you provided in combination with a fresh RabbitMQ 3.8.11. To fix it, I had to download the Let’s Encrypt R3 and ISRG Root X1 certificate PEM files and add them to the top of the fullchain1.pem file (attached, https://letsencrypt.org/certificates/).

I used the attached rabbitmq.conf file as well. Note that I commented out some settings you were using and added the "depth" setting.

I tested the connection with this command:

openssl s_client -connect localhost:5671 -cert cert1.pem -key privkey1.pem -CAfile fullchain1.pem -verify 8 -verify_hostname invalidtest.mohag.za.net

I started RabbitMQ using the "generic-unix" package and Erlang 23.2.3 with this command, after adding invalidtest.mohag.za.net to my /etc/hosts file as an entry for 127.0.0.1:

RABBITMQ_CONFIG_FILE="/home/lbakken/issues/rabbitmq-users/tls-connections-dropping-lquxjze8UIc/rabbitmq.conf" RABBITMQ_NODENAME=rab...@invalidtest.mohag.za.net RABBITMQ_USE_LONGNAME=true LOG=debug RABBITMQ_ALLOW_INPUT=true ./sbin/rabbitmq-server

I think in your scenario the root cause is the missing root certificate files. The Erlang VM has no knowledge of system cert stores.

Thanks,

Luke

[Gert van den Berg]

On Saturday, January 30, 2021 at 6:18:51 PM UTC+2 Luke Bakken wrote:

Hi Gert,

Can you confirm that this issue only happens after an upgrade? Do you see the issue with a 100% fresh installation?

I'm seeing it on a new install, as well as on an upgrade to 3.8.10 / 3.8.11 from an earlier version. (3.8.9 is known to work) 

[Luke Bakken]

Thanks Gert,

Please see this message where I resolve your issue by including the full cert chain in your cacertfile file:

https://groups.google.com/g/rabbitmq-users/c/lquxjze8UIc/m/9E5mkg88AgAJ

[Gert van den Berg]

On Sunday, January 31, 2021 at 6:18:39 PM UTC+2 Luke Bakken wrote:

Thanks Gert,

Please see this message where I resolve your issue by including the full cert chain in your cacertfile file:

https://groups.google.com/g/rabbitmq-users/c/lquxjze8UIc/m/9E5mkg88AgAJ

There was a response on that, that seems to be gone now somehow. We are not using client certificates (SASL PLAIN authentication is used over the TLS connections). Your tests show client certificates being used on s_cllient. (and the depth setting seems to related to client certificate validation)

 Is the full chain up to the root certificates needed for the server certificate? (AFAIK, it should have the host's certificate and enough intermediate certificates (sometimes called "chain certificates") that the client can validate the server against its certificate store? (The server would often not be able to verify the certificate (except against the CA, since it would often be self-signed)

(This happens on all TLS ports, including the management interface and MQTTS)

(3.8.9 sends the server certificate, 3.8.10 and 3.8.11 does not) (on a TLS connection without a client certificate)

Gert

[Gert van den Berg]

If I disable TLS v1.3 in the config file (by commenting it out) (I used the Vagrant one to test) s_client receives a certificate again. (I did that on both the management plugin and the normal AMQP / MQTT settings)

This mostly makes sense, since 3.8.10 removed TLS v1.3 from the version blacklist... https://github.com/rabbitmq/rabbitmq-common/pull/416

If I force it to use TLS v1.2 with the config as in the Vagrantfile, it also works. Forcing TLS v1.3 results in no certificate being received: (This is with the config as in the Vagrantfile on RabbitMQ 3.8.11)

root@ubuntu-18:~# openssl s_client -connect 127.0.0.1:5671 -tls1_2  # Working - forced to TLS v1.2

CONNECTED(00000005)

depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3

verify return:1

depth=1 C = US, O = Let's Encrypt, CN = R3

verify return:1

depth=0 CN = invalidtest.mohag.za.net

verify return:1

---

Certificate chain

 0 s:CN = invalidtest.mohag.za.net

   i:C = US, O = Let's Encrypt, CN = R3

 1 s:C = US, O = Let's Encrypt, CN = R3

   i:O = Digital Signature Trust Co., CN = DST Root CA X3

<snip some output>

---

SSL handshake has read 3137 bytes and written 418 bytes

Verification: OK

---

New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384

Server public key is 2048 bit

Secure Renegotiation IS supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

SSL-Session:

    Protocol  : TLSv1.2

    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

    Session-ID: 0CED0548D5CE215228D9AC00115A279ECE665851A2B093018BFB2ED9925520D0

    Session-ID-ctx:

    Master-Key: 840EAA64069ADA12AE8CC52ED343808C0CF60680F5A6045A0F6815DC427ED27B2FEDC2489FEF7E334F5929D069BD6118

    PSK identity: None

    PSK identity hint: None

    SRP username: None

    Start Time: 1612173114

    Timeout   : 7200 (sec)

    Verify return code: 0 (ok)

    Extended master secret: no

---

==========================================

root@ubuntu-18:~# openssl s_client -connect 127.0.0.1:5671 -tls1_3 # Not working - forced to TLS v1.3

CONNECTED(00000005)

write:errno=104

---

no peer certificate available

---

No client certificate CA names sent

---

SSL handshake has read 0 bytes and written 233 bytes

Verification: OK

---

New, (NONE), Cipher is (NONE)

Secure Renegotiation IS NOT supported

Compression: NONE

Expansion: NONE

No ALPN negotiated

Early data was not sent

Verify return code: 0 (ok)

---

[Gert van den Berg]

On Monday, February 1, 2021 at 11:57:13 AM UTC+2 Gert van den Berg wrote:

If I disable TLS v1.3 in the config file (by commenting it out) (I used the Vagrant one to test) s_client receives a certificate again. (I did that on both the management plugin and the normal AMQP / MQTT settings)

This mostly makes sense, since 3.8.10 removed TLS v1.3 from the version blacklist... https://github.com/rabbitmq/rabbitmq-common/pull/416

If I force it to use TLS v1.2 with the config as in the Vagrantfile, it also works. Forcing TLS v1.3 results in no certificate being received: (This is with the config as in the Vagrantfile on RabbitMQ 3.8.11)

If I remove the cipher list and TLS v1.2, RabbitMQ fails to start (with just TLS v1.3 enabled with the default ciphers) (I were trying to test whether the list of ciphers that I have, might not contain any TLS v1.3 ciphers)

With both TLS v1.2 and TLS v1.3 enabled and a cipher list not supplied, I can connect using TLS v1.3 (The cipher list seems to be part of the problem) (I'm not completely sure if it lacks any TLS v1.3 ciphers or if there is another issues)

(I'm not having luck running it with only TLS v1.3  though, which might also be (a different issue)) (I need TLSv1.2 as well, but it might be worth looking into)

[Gert van den Berg]

It seems like the issue here is that 3.8.10 enabled TLS v1.3 (It was previously disabled even if listed) and my config lacking any TLS v1.3 ciphers.

With this cipher list, it works (and pass testssl.sh's checks) on the newer versions: (similar list needed for management interface and prometheus TLS settings, if configured):

# TLS v1.3 ciphers

ssl_options.ciphers.1 = TLS_AES_256_GCM_SHA384

ssl_options.ciphers.2 = TLS_AES_128_GCM_SHA256

ssl_options.ciphers.3 = TLS_CHACHA20_POLY1305_SHA256

# TLS v1.2 ciphers

ssl_options.ciphers.4 = ECDHE-ECDSA-AES256-GCM-SHA384

ssl_options.ciphers.5 = ECDHE-RSA-AES256-GCM-SHA384

ssl_options.ciphers.6 = ECDHE-ECDSA-CHACHA20-POLY1305

ssl_options.ciphers.7 = ECDHE-RSA-CHACHA20-POLY1305

ssl_options.ciphers.8 = ECDHE-ECDSA-AES128-GCM-SHA256

ssl_options.ciphers.9 = ECDHE-RSA-AES128-GCM-SHA256

ssl_options.ciphers.10 = DHE-RSA-AES256-GCM-SHA384

ssl_options.ciphers.11 = DHE-RSA-AES128-GCM-SHA256

ssl_options.ciphers.12 = DHE-RSA-CHACHA20-POLY1305

ssl_options.ciphers.13 = ECDH-ECDSA-AES256-GCM-SHA384

ssl_options.ciphers.14 = ECDH-RSA-AES256-GCM-SHA384

ssl_options.ciphers.15 = ECDH-ECDSA-AES128-GCM-SHA256

ssl_options.ciphers.16 = ECDH-RSA-AES128-GCM-SHA256

Thanks for the assistance.

[Luke Bakken]

Yes, the full chain is needed because the Erlang VM has no knowledge of any system certificate stores (as far as I know).

[Luke Bakken]

Hi Gert,

Nice work figuring that out. I had to assist a customer last week with exactly the same issue and didn't think about your case as well. They did not have the necessary ciphers enabled. I traced their error to this code, which at least has a helpful comment:

https://github.com/erlang/otp/blob/master/lib/ssl/src/tls_record.erl#L422-L432

Thanks,

Luke