RabbitMQ error for MTLS connection from client to server.

[Benjamin Wilson]

Hi,

I am running into an odd issue with tls connections from client to server. Openssl connection to opennssl successfully completes the tls handshake. If I run openssl client from local host I also have no issues. I generated my certificates from a certificate authority I set up on a windows server. configs for rabbitmq are set up to accept all connections for now and not refuse the certs.  I noticed I get an errno 104 error as soon as rabbitmq servers and I try to connect using the openssl command over port 5761.  

Below is the errors I am seeing in the logs. It looks like it is reaching the remote host but the handshake is not happening.

7:51:03.900507+00:00 [error] <0.9344.0>     initial call: rabbit_reader:init/3
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     pid: <0.9344.0>
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     registered_name: []
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     exception exit: {{{error,{"evp.c",136},"Can't make context"},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                       [{crypto,generate_key,3,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"crypto.erl"},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                             {line,1690},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                             {error_info,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                                 #{erl_function_arg_num => undefined}}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {ssl_cipher,generate_server_share,1,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"ssl_cipher.erl"},{line,1160}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {tls_server_connection_1_3,do_handle_client_hello,2,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"tls_server_connection_1_3.erl"},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                             {line,459}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {tls_server_connection_1_3,handle_client_hello,2,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"tls_server_connection_1_3.erl"},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                             {line,381}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {gen_statem,loop_state_callback,11,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"gen_statem.erl"},{line,1395}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {tls_connection,init,1,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"tls_connection.erl"},{line,162}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                        {proc_lib,init_p_do_apply,3,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                            [{file,"proc_lib.erl"},{line,241}]}]},
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                      {gen_statem,call,[<0.9341.0>,{start,5000},infinity]}}
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in function  gen:do_call/4 (gen.erl, line 246)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from gen_statem:call/3 (gen_statem.erl, line 923)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from ssl_gen_statem:call/2 (ssl_gen_statem.erl, line 1319)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from ssl_gen_statem:handshake/2 (ssl_gen_statem.erl, line 253)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from ranch_ssl:handshake/3 (src/ranch_ssl.erl, line 180)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from ranch:handshake1/2 (src/ranch.erl, line 281)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from rabbit_networking:handshake/2 (rabbit_networking.erl, line 591)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>       in call from rabbit_reader:init/3 (rabbit_reader.erl, line 164)
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     ancestors: [<0.9342.0>,<0.760.0>,<0.759.0>,<0.758.0>,<0.756.0>,
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>                   <0.755.0>,rabbit_sup,<0.250.0>]
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     message_queue_len: 0
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     messages: []
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     links: [<0.9342.0>]
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     dictionary: []
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     trap_exit: false
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     status: running
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     heap_size: 987
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     stack_size: 28
2025-03-13 17:51:03.900507+00:00 [error] <0.9344.0>     reductions: 223

[Luke Bakken]

Hello,

Just to set expectations, I'll point out Team RabbitMQ's community support policy - https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md#who-is-eligible-for-community-support

In general, we do not diagnose TLS related issues for the community. Since you mentioned Windows, I thought I would get some essential information you left out:

• Windows version and patch level
• RabbitMQ version
• Erlang version
• Are you running RabbitMQ on Windows?
• Exact openssl commands used
• Full RabbitMQ configuration file
• Are you trying to use FIPS?

"Can't make context" is not an error I've ever seen, and it is very rare - https://www.google.com/search?q=erlang+%22Can%27t+make+context%22

You can see from the stack trace that a TLS 1.3 negotiation fails. RabbitMQ on Windows still uses OpenSSL underneath, not the Windows SChannel library, so there shouldn't be an issue like this.

Thanks,

Luke

[Benjamin Wilson]

Hi Luke,

To add more context. Windows Server 2022 is the CA issuing out certs.

I have rabbitmq-server-3.13.1-1  running on a DISA Stig'd RHEL 8 machine version 8.10.

exact ssl command is.

openssl s_client -connect localhost:8443 -cert /etc/ssl/groundsrv.pem -key /etc/ssl/groundkey.key -CAfile /etc/ssl/groundca.pem -verify 8 -verify_hostname groundstation-ldap

which worked on local host and port 5671 as well.

then tried from client vi 5671 since I setup tls on this port.

openssl s_client -connect (ip address of host):5671 -cert /etc/ssl/groundsrv.pem -key /etc/ssl/groundkey.key -CAfile /etc/ssl/groundca.pem -verify 8 -verify_hostname groundstation-ldap

The management UI works great with no issues when using ssl.

We are not trying to use FIPS.

What is the cost for support if we needed it?

listeners.tcp.default = 127.0.0.1:5672

listeners.tcp.other = 192.168.0.10:5672

# SSL configuration

listeners.ssl.default = 5775

listeners.ssl.default = 127.0.0.1:5775

listeners.ssl.other = 192.168.0.10:5775

 ssl_options.cacertfile = /etc/ssl/groundca.pem

 ssl_options.certfile   = /etc/ssl/groundsrv.pem

 ssl_options.keyfile    = /etc/ssl/groundkey.pem

# ssl_options.verify     = verify_none

# # ssl_options.fail_if_no_peer_cert = true

# ssl_options.fail_if_no_peer_cert = false

management UI configuration

 management.tcp.port = 15672

 management.ssl.port       = 15671

 management.ssl.cacertfile = /etc/ssl/groundca.pem

 management.ssl.certfile   = /etc/ssl/groundsrv.pem

 management.ssl.keyfile    = /etc/ssl/groundkey.pem

# management.ssl.honor_cipher_order   = true

# management.ssl.honor_ecc_order      = true

# management.ssl.client_renegotiation = false

# management.ssl.secure_renegotiate   = true

#management.ssl.versions.1 = tlsv1.2

#management.ssl.ssl_options.ciphers = TLS_AES_256_GCM_SHA384

#management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384

#management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384

#management.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384

#management.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384

#management.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384

#management.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384

#management.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384

#management.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384

#management.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384

[Benjamin Wilson]

I meant to say we are using port 5775 when testing the ssl connections. Just to not add confusion

[Luke Bakken]

What does "DISA Stig'd" mean?

[Benjamin Wilson]

it is a security technical implementation guide which is followed in order to secure a system. Usually on the DoD side. It is used to harden the system down and make it more secure.

[Luke Bakken]

OK, that sounds like something that could interfere with TLS, potentially. Has TLS ever worked in an environment like this for you?

[Benjamin Wilson]

Yeah so it looks like I got the connection working after disabling FIPS on the RHEL 8 server end. Is there a way to modify the evc.p file to use fips approved alogirthms/key exchanges or do we need to specifically use X25519 and X448.
These are the 3 I am seeing as approved and supported in openssl libraries

 P-256 (secp256r1)

P-384 (secp384r1)

P-521 (secp521r1)

[Luke Bakken]

evp.c is a source file within the OpenSSL library, which we don't have any control over.

Open-Source RabbitMQ and Erlang do support FIPS, however, it is not something for which Team RabbitMQ provides free community support. Perhaps another user who uses FIPS will chime in here.

Otherwise, this is the information for obtaining a support contract -

https://www.rabbitmq.com/contact#tanzu-rabbitmq

[Benjamin Wilson]

Looks like we figured it out. We had to use TLS1.2 changed the advanced configs to turn FIPS on then put in the rabbitmq.conf to use approved ciphers that are FIPS compliant.

[Luke Bakken]

Thank you for reporting back!