RabbitMQ/MassTransit TLS instant handshake timeout

[Harry Young]

We are having an issue where 2 very specific servers seem to have trouble establishing a TLS connection with our instance of RabbitMQ. The clients are .NET projects that connect to Rabbit via MassTransit. We are certain that the issue isn't code related as the exact same package is deployed and installed on each server. Talking to the network engineers, we were told all the servers involved (Working and not working) were created off the exact same image and there is no firewall sat between the servers in question as it is all internal app servers. 

On the working servers, the apps connect fine, and the connection appears in the RabbitMQ connection tab, for the servers that don’t work, we get the error in the program:

Service cannot be started. MassTransit.RabbitMqTransport.RabbitMqConnectionException: Connect failed: DfSTGRabbitMQ@dfstgapp01.************.co.uk:5671/stg ---> RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Security._SslStream.StartWriting(Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security._SslStream.ProcessWrite(Byte[] buffer, Int32 offset, Int32 cou...

Looking at the RabbitMQ logs, this all we get when the apps attempt to connect to Rabbit:

2018-07-20 16:08:08.011 [info] <0.2236.0> accepting AMQP connection <0.2236.0> (10.1.123.102:60341 -> 10.1.123.102:5671)
2018-07-20 16:08:08.011 [warning] <0.2236.0> closing AMQP connection <0.2236.0> (10.1.123.102:60341 -> 10.1.123.102:5671):
{handshake_timeout,handshake}

We are very confused as the servers and program is identical but one server it connects fine and the other it can’t seem to establish a TLS connection (Non TLS connections connect just fine). Has anybody else run into this issue and could help? Any settings we may have missed or something we need to look at?

Thanks

[Michael Klishin]

There are two doc guides available to assist you in such scenarios [1][2]. The log suggests

it's an AMQP 0-9-1 handshake that times out [4], not a TLS upgrade.

I'd recommend proceeding with a traffic capture collected [3].

That's about as much as can be suggested without having any RabbitMQ, Erlang or effective configuration [5]

information.

1. http://www.rabbitmq.com/troubleshooting-ssl.html

2. https://www.rabbitmq.com/troubleshooting-networking.html

3. http://www.rabbitmq.com/amqp-wireshark.html

4. https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/rabbitmq.conf.example#L44

5. http://www.rabbitmq.com/configure.html#verify-configuration-config-file-location

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Harry Young]

Hi, thanks for your response.

I've been trying to use Wireshark to capture the AMQP traffic but there doesn't seem to be any useful information coming up.

I don't see any traffic under "AMQP" protocol but can see some connections going to the right port but just under "TCP".

I have followed the guide you have linked but still no joy and have ensured to download the latest version of Wireshark.

Also is there any configuration in particular that I should share that might shed more light on what the issues we are having are?

Thanks

Harry

[Michael Klishin]

You are likely capturing on the wrong interface or use a non-standard port for AMQP 0-9-1/1.0 clients.

Wireshark operates at the networking protocol level and requires no special

RabbitMQ configuration except when TLS is used. In that case Wireshark must be configured to use a private key that would

allow it to decrypt connection traffic.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.