.NET Client API -> Unexpected Connection / Channel Shutdowns

[Jason Ipock]

I have a service that is creating a RabbitMQ Consumer that occasionally experiences a connection / channel(model) shutdown event. It is a single consumer, running on its own connection and channel,  connecting  to a quorum queue spread among three RabbitMQ servers. I haven't found a reproducible scenario for this, except for just waiting. It has happened multiple times over the last week.  I have done some logging and I'll include it as text at the end of this message. 

The message returned says that it might be an authentication issue. This consumer service seems to have no authentication issue starting up and connecting to the specified RabbitMQ queue. There does not seem to be any in the Windows Event Logs which indicate a problem (reboots, system service failures, etc.), although I only searched for things I thought would obviously cause an authentication / network failure. The only thing of note is that on one of the three RabbitMQ servers hosting the Quorum queue is logging a failed heartbeat check at approximately the same time. This is not noted in either of the other two servers.

Has anyone either

* Experienced this issue

* Has an idea on how to address the root cause of the issue

* Has different logging suggestions

* or has a workaround besides simply restarting the service?

Much thanks to all who read this. The logs follow.....

2021-09-20 11:27:16,515 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - ConnectionShutdown called

2021-09-20 11:27:16,517 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> ChannelMax: 2047

2021-09-20 11:27:16,525 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Close Reason: {"Cause":{"ClassName":"System.IO.EndOfStreamException","Message":"Reached the end of the stream. Possible authentication failure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)\r\n   at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024858,"Source":"RabbitMQ.Client","WatsonBuckets":null},"ClassId":0,"Initiator":1,"MethodId":0,"ReplyCode":0,"ReplyText":"End of stream"}

2021-09-20 11:27:16,529 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Client Properties: {"product":"UmFiYml0TVE=","version":"Ni4yLjErMDFjMDdjODdmMTkwMTliMzBmMzY4ZGIzNTc1ZDA2Njg1ODM4NzA3Nw==","platform":"Lk5FVA==","copyright":"Q29weXJpZ2h0IChjKSAyMDA3LTIwMjAgVk13YXJlLCBJbmMu","information":"TGljZW5zZWQgdW5kZXIgdGhlIE1QTC4gU2VlIGh0dHBzOi8vd3d3LnJhYmJpdG1xLmNvbS8=","capabilities":{"publisher_confirms":true,"exchange_exchange_bindings":true,"basic.nack":true,"consumer_cancel_notify":true,"connection.blocked":true,"authentication_failure_close":true},"connection_name":"My_Service - Server4"}

2021-09-20 11:27:16,529 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Server Properties: {"capabilities":{"publisher_confirms":true,"exchange_exchange_bindings":true,"basic.nack":true,"consumer_cancel_notify":true,"connection.blocked":true,"consumer_priorities":true,"authentication_failure_close":true,"per_consumer_qos":true,"direct_reply_to":true},"cluster_name":"cHJvZHVjdGlvbl9jbHVzdGVyQHJhYmJpdE1R","copyright":"Q29weXJpZ2h0IChjKSAyMDA3LTIwMjEgVk13YXJlLCBJbmMuIG9yIGl0cyBhZmZpbGlhdGVzLg==","information":"TGljZW5zZWQgdW5kZXIgdGhlIE1QTCAyLjAuIFdlYnNpdGU6IGh0dHBzOi8vcmFiYml0bXEuY29t","platform":"RXJsYW5nL09UUCAyMy4yLjQ=","product":"UmFiYml0TVE=","version":"My45LjE="}

2021-09-20 11:27:16,529 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> IsOpen: No

2021-09-20 11:27:16,531 [4] DEBUG (DomainNamespace)  .RabbitMQConsumer - Event Arg Data: {"Cause":{"ClassName":"System.IO.EndOfStreamException","Message":"Reached the end of the stream. Possible authentication failure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)\r\n   at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024858,"Source":"RabbitMQ.Client","WatsonBuckets":null},"ClassId":0,"Initiator":1,"MethodId":0,"ReplyCode":0,"ReplyText":"End of stream"}

2021-09-20 11:27:16,534 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - ModelShutdown called

2021-09-20 11:27:16,534 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Channel: 1

2021-09-20 11:27:16,535 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Is Open: No

2021-09-20 11:27:16,535 [4] DEBUG  (DomainNamespace)  .RabbitMQConsumer - -----> Is Closed: Yes

2021-09-20 11:27:16,535 [4] DEBUG (DomainNamespace).RabbitMQConsumer - -----> Close Reason: {"Cause":{"ClassName":"System.IO.EndOfStreamException","Message":"Reached the end of the stream. Possible authentication failure.","Data":null,"InnerException":null,"HelpURL":null,"StackTraceString":"   at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(Stream reader, Byte[] frameHeaderBuffer)\r\n   at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration()\r\n   at RabbitMQ.Client.Framing.Impl.Connection.MainLoop()","RemoteStackTraceString":null,"RemoteStackIndex":0,"ExceptionMethod":null,"HResult":-2147024858,"Source":"RabbitMQ.Client","WatsonBuckets":null},"ClassId":0,"Initiator":1,"MethodId":0,"ReplyCode":0,"ReplyText":"End of stream"}

[Jason Ipock]

Almost forgot the system info: 

RabbitMQ 3.9.1

Erlang 23.2.4

I should also note that this is NOT a high-volume system to say the least. This queue might have a couple of messages a minute. And in the evening / night hours, none at all. The error logged was during the daytime when there is an average amount of use. 

[Jaco De Villiers]

Hi, Did you manage to resolve the problem?  I have exactly the same problem, the producer just stops sending messages and I receive:

 Failed to publish 2 - Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text='End of stream', classId=0, methodId=0, cause=System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.

[Luke Bakken]

Hello

On Thursday, September 1, 2022 at 8:38:46 AM UTC-7 jac...@gmail.com wrote:

Hi, Did you manage to resolve the problem?  I have exactly the same problem, the producer just stops sending messages and I receive:

 Failed to publish 2 - Already closed: The AMQP operation was interrupted: AMQP close-reason, initiated by Library, code=0, text='End of stream', classId=0, methodId=0, cause=System.IO.EndOfStreamException: Reached the end of the stream. Possible authentication failure.

 

Rather than replying to such an old discussion, we recommend starting a new one. Please include more information:

• .NET version
• RabbitMQ .NET client version
• RabbitMQ version
• Erlang version
• Server and client operating systems
• Logs from RabbitMQ, especially at the same time(s) you see the exception!

Without that information it is impossible to assist.

Thanks,

Luke

[Jaco De Villiers]

Thank you, I thought continuation on a thread was preferable, my mistake.

I am creating a new thread

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/b2HUHoniU5A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ab7d68a2-9791-4250-908c-8f13ef4ca7a3n%40googlegroups.com.