Troubleshooting 541 Internal Errors

[Ryan S. Barton]

Greetings,

We have an application utilizing a combination of MassTransit 2.9 and the RabbitMQ .NET client to connect to a three node RabbitMQ 3.2.4 cluster sitting behind an F5 load balancer.  Intermittently (the best kind), the MassTransit-based clients will log an error saying that they've had a connection reset:

[ServiceBusReceiveContext: ERROR] - Consumer Exception ExposedMassTransit.Transports.InvalidConnectionException: rabbitmq://rabbitmq/_Terminator_Publisher_64 => Invalid connection to host ---> RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=541, text="INTERNAL_ERROR", classId=0, methodId=0, cause=

From what I've been able to gather via The Google, it seems that an INTERNAL_ERROR / 541 should probably result in some sort of error in the RabbitMQ logs -- but there are no errors to match in our RabbitMQ server logs.

Are there additional log levels we should be looking at?  Our current log level is the default (connection and info).

Many thanks,

-rsb

[Simon MacMullen]

On 09/01/15 17:09, Ryan S. Barton wrote:
> From what I've been able to gather via The Google, it seems that an
> INTERNAL_ERROR / 541 should probably result in some sort of error in the
> RabbitMQ logs -- but there are no errors to match in our RabbitMQ server
> logs.

Yes, absolutely. If it's coming from the server it will get logged. Have
you looked in the -sasl log as well? (Mind you, 541 internal error
should get logged in the main log).

> Are there additional log levels we should be looking at? Our current
> log level is the default (connection and info).

You don't need to enable anything to log internal errors.

Cheers, Simon

[Ryan S. Barton]

Greetings,

Yes -- I have scrubbed through both sets of logs on each of our servers.  The -sasl logs are pretty sparse on this cluster (it's not heavily used), making it easy to correlate the times.

Because only our MassTransit clients were seeing this error, my next stop was the MassTransit users list -- I'm presuming that MassTransit is faithfully passing along an error coming from the broker, but maybe it's being....creative?....or something.

Thanks,

-rsb