.NET Client cannot connect to to AMQP over TLS on Port 5671 Windows Server 2019 Standard version 1809

[Никола Калфов]

I am using a Windows Server 2019 Standard Version 1809 Build 17763.2803

I have first installed Erlang 26.0 (otp_win64_26.0.exe). I have added Handle.exe to PATH. I have also added ERLANG_HOME (value C:\Program Files\Erlang OTP)

Then I have installed RabbitMQ 3.12.1 (rabbitmq-server-3.12.1.exe). I have successfully connected to port 5672 with a .NET Client with NuGet RabbitMQ.Client 6.2.1

Then I create server certificate using tls-gen:

make PASSWORD=12345678 CN=machine.domain.local DAYS_OF_VALIDITY=3650 NUMBER_OF_PRIVATE_KEY_BITS=2048 CLIENT_ALT_NAME=machine.domain.local

I am installing cacert.cer from tls-gen\basic\testca

Then I am taking server certificate, key and ca_certificate.pem from tls-gen\basic\result and configuring them. I tried adding the password and removing it, whatever else, but for example:

[{rabbit, [ {log_levels, [{connection, debug}, {queue, debug}]}, {log, [ {file, [ {level, debug} ] }, {categories, [ {connection,[{level, debug}]}, {queue, [{level, debug}]}, {console, [{enabled, true},{level, debug}]} ] } ] }, {ssl_listeners, [5671]}, {ssl_options, [ {cacertfile, "C:\\Program Files\\RabbitMQ Server\\rabbitmq_server-3.12.1\\certs\\ca_certificate.pem"}, {certfile, "C:\\Program Files\\RabbitMQ Server\\rabbitmq_server-3.12.1\\certs\\server_certificate.pem"}, {keyfile, "C:\\Program Files\\RabbitMQ Server\\rabbitmq_server-3.12.1\\certs\\server_key.pem"}, {password, "12345678"} ] } ] }].

The client fails to connect to RabbitMQ over TLS. It throws exception:

RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.AggregateException: One or more errors occurred. (Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host..) ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. ---> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.

I can see rabbitmq is listening on the port, however I cannot connect to it:

2023-06-30 18:47:28.888000+03:00 [info] <0.692.0> started TLS (SSL) listener on [::]:5671 2023-06-30 18:47:28.888000+03:00 [info] <0.712.0> started TLS (SSL) listener on 0.0.0.0:5671 2023-06-30 18:47:28.935000+03:00 [debug] <0.543.0> Plugins discovery: ignoring getopt, not a RabbitMQ plugin 2023-06-30 18:47:28.935000+03:00 [debug] <0.543.0> Plugins discovery: ignoring quantile_estimator, not a RabbitMQ plugin 2023-06-30 18:47:28.997000+03:00 [debug] <0.543.0> Plugins discovery: ignoring getopt, not a RabbitMQ plugin 2023-06-30 18:47:28.997000+03:00 [debug] <0.543.0> Plugins discovery: ignoring quantile_estimator, not a RabbitMQ plugin 2023-06-30 18:47:29.029000+03:00 [info] <0.543.0> Server startup complete; 3 plugins started. 2023-06-30 18:47:29.029000+03:00 [info] <0.543.0> * rabbitmq_management 2023-06-30 18:47:29.029000+03:00 [info] <0.543.0> * rabbitmq_web_dispatch 2023-06-30 18:47:29.029000+03:00 [info] <0.543.0> * rabbitmq_management_agent 2023-06-30 18:47:29.029000+03:00 [debug] <0.543.0> Marking RabbitMQ as running 2023-06-30 18:47:29.029000+03:00 [debug] <0.543.0> Change boot state to `ready` 2023-06-30 18:47:29.029000+03:00 [debug] <0.148.0> Boot state/systemd: notifying of state `ready` 2023-06-30 18:47:29.232000+03:00 [debug] <0.9.0> Time to start RabbitMQ: 5793485 us 2023-06-30 18:47:35.164000+03:00 [debug] <0.719.0> Plugins discovery: ignoring getopt, not a RabbitMQ plugin 2023-06-30 18:47:35.164000+03:00 [debug] <0.719.0> Plugins discovery: ignoring quantile_estimator, not a RabbitMQ plugin 2023-06-30 18:48:13.260000+03:00 [notice] <0.724.0> TLS server: In state hello at tls_handshake.erl:354 generated SERVER ALERT: Fatal - Insufficient Security 2023-06-30 18:48:13.260000+03:00 [notice] <0.724.0> - no_suitable_ciphers

On the server I have forbid all older TLS versions. Only TLSv1.2 is active and for now I cannot support TLSv1.3. I have allowed all of the possible cypher suites using IISCrypto. I am not certain if the issue is in the TLS Handshake due to the client and server trying to negotiate something they can't or the certificate is invalid (tls-gen says it is valid). I have done this a couple of times on Windows Server 2012 without any issues. I even tried copying certificate that is working fine from another machines, but I keep getting the error.

[Nikola Kalfov]

I've disabled TLSv1.3 in RabbitMQ config and allowed only TLSv1.2 and the issue was resolved.

[Luke Bakken]

Thanks for taking the time to report this.

I will see if I can reproduce " no_suitable_ciphers" using my Windows 11 laptop.

I'm assuming your C# client code isn't doing anything special at this time, just trying to open a TLS connection to RabbitMQ.

Thanks,

Luke

[Luke Bakken]

I'll continue the discussion here, but I'm providing this link for reference - https://stackoverflow.com/questions/76590624/net-client-cannot-connect-to-to-amqp-over-tls-on-port-5671-windows-server-2019/

[Nikola Kalfov]

Hi Luke,

The server doesn't support TLSv1.3. Due to limitations in the infrastructure, both RabbitMQ and the services are installed on Windows Server 2019. The clients are developed with .NET 5 and RabbitMQ.Client 6.2.1, which may support TLSv1.3, but their host doesn't. The clients are trying to handshake on TLSv1.3 with RabbitMQ and fail. I have tried updating RabbitMQ.Client to 6.5.0, but the issue continued to manifest.

It seems openssl s_client can negotiate TLSv1.3 to the RabbitMQ instance. As a workaround, I have explicitly stated to RabbitMQ to support only TLSv1.2 and now the clients connect successfully. It seems something doesn't respect that TLSv1.3 could not be established due to the underlying infrastructure and tries to negotiate it.

I haven't set cyphers as shown in https://www.rabbitmq.com/ssl.html#tls-versions-why-not-limit and probably I will not need to until the servers are updated to Windows 2022 (or changed to gnu/linux).

[Luke Bakken]

Hi Nikola,

I appreciate the follow-up! I see, it's a limitation in Windows Server 2019 TLS implementation.