RabbitMQ Super Old Version Seeing Code 541 errors

[Eric Aldinger]

We are using Rabbit MQ on Windows. We have started seeing code 541 errors with no known changes in code or broker configuration. Please pity me. 

Short of upgrading, do any of you know how to resolve this? It takes two weeks to upgrade ANYTHING here. We have heartbeats set to 10 (also tested with 60). We use BasicPut and BasicConsume. 

ERROR

2017-04-19 04:27:18,674 - ERROR - Could not connect to Rabbit: AuthenticationMechanism Plain, AutomaticRecoveryEnabled True, HostName <REMOVED>, Port 5672, Heartbeat 10, VirtualHost /

RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

VERSIONS

rabbitmq_server-3.2.4
Microsoft Windows Server 2012 R2 Datacenter
erl5.10.4
C# rabbitmq client version="3.5.6" targetFramework="net45"
.net client versions installed
Client 4.6.01055
Full 4.6.01055
Our network supports IPV4 only

[Michael Klishin]

See server logs for clues.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Eric Aldinger]

Sorry. The error from the application logs seems to echo our application logs, could not connect. However we are seeing the broker become non-responsive and the management console also stops responding (seeming to confirm the issue is at the broker level). 

Rabbit log

=ERROR REPORT==== 24-Apr-2017::17:52:15 ===

closing AMQP connection <0.1519.0> (172.22.10.166:50605 -> 172.22.10.166:5672):

{handshake_timeout,frame_header}

On Sunday, April 23, 2017 at 2:04:09 AM UTC-7, Michael Klishin wrote:

See server logs for clues.

On Sun, Apr 23, 2017 at 3:30 AM, Eric Aldinger <ewas...@gmail.com> wrote:

We are using Rabbit MQ on Windows. We have started seeing code 541 errors with no known changes in code or broker configuration. Please pity me. 

Short of upgrading, do any of you know how to resolve this? It takes two weeks to upgrade ANYTHING here. We have heartbeats set to 10 (also tested with 60). We use BasicPut and BasicConsume. 

ERROR

2017-04-19 04:27:18,674 - ERROR - Could not connect to Rabbit: AuthenticationMechanism Plain, AutomaticRecoveryEnabled True, HostName <REMOVED>, Port 5672, Heartbeat 10, VirtualHost /

RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

VERSIONS

rabbitmq_server-3.2.4
Microsoft Windows Server 2012 R2 Datacenter
erl5.10.4
C# rabbitmq client version="3.5.6" targetFramework="net45"
.net client versions installed
Client 4.6.01055
Full 4.6.01055
Our network supports IPV4 only

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Surely there is something else in the logs (one thing I'm certain about is that those 541 exceptions will be logged, in the -sasl log if not the regular one).

Consider posting the entire log for the entire day (12 hours before and after) when that happened.

"Handshake timeout" means that a TCP connection was accepted from the client but AMQP 0-9-1 handshake never proceeded past

a certain point. It's not a root cause of anything.

Regardless, please upgrade. You don't have to upgrade the .NET client at the same time, an older

version will work just fine.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Aldinger]

I would post it if there was. I get a client error in our application logs, a hand shake broker error in our rabbit log, and nothing in the event logs for the server. I have NEVER seen this and I am at a loss. We did not see any port exhaustion issues when running netstat. Memory pressure is not high. CPU is fine. I do not have a complete wireshark or procman log, but I have recommended those for our IT staff. Am I missing some sort of erlang logging?

RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.AlreadyClosedException: Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=541, text="Unexpected Exception", classId=0, methodId=0, cause=System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

And a broker error

=INFO REPORT==== 24-Apr-2017::19:56:17 ===

    LDAP DECISION: does {REDACTED} have tag management? true

=INFO REPORT==== 24-Apr-2017::19:56:17 ===

LDAP DECISION: login for {REDACTED}0: ok

=ERROR REPORT==== 24-Apr-2017::19:56:17 ===

closing AMQP connection <0.31759.0> (172.22.10.175:56417 -> 172.22.10.175:5672):

{handshake_timeout,frame_header}

[Michael Klishin]

Eric,

There are two logs: regular and -sasl, the latter only contains unhandled exceptions.

Are you suggesting that

 * The sasl log is empty

and

 * You only have 1 second worth of logs in the regular log?

I recall there was a 3.2.x release that fixed a bug that could prevent all logging

of unhandled exceptions or possibly more. Unless that's the exact issue you are hitting,

all issues that result in an 541 Internal Error response should be logged.

We need *the entire* log for, say, 24 around the event. A few entries around

one client connections are not sufficient and will not hint at the root cause in all likelihood.

And I'd like to reiterate that you are close to 30 releases behind:

http://www.rabbitmq.com/changelog.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Eric Aldinger]

SASL log is empty when these events occur (we do hourly log rolling). I do see entries when we restart the service.

I pasted the immediate context for the error. It is a bunch of active directory auth negotiation info messages based on our rabbitmq security configuration. I think we can close this thread out. We will upgrade the RabbitMQ broker. Hopefully that will fix the issue or log the errors more clearly.

[Michael Klishin]

If you provided a least a few hours worth of logs maybe we could trace it to a specific

known issue. But it's up to you and upgrading is a good idea.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.