Problem with Spring-amqp, listener stops with 'Consumer received fatal exception on startup',

[Wim Veldhuis]

Hi,

We are running a number of services using AMQP as messaging middleware. We quite regularly have to restart services as they become unresponsive to requests from clients.

We see the following in the logfile:

;2015-05-08 00:35:54,578; DEBUG; servicename=; clusterid=; username=; msguuid=; org.springframework.amqp.rabbit.core.RabbitAdmin - Declarations finished

;2015-05-08 00:37:12,015; ERROR; servicename=; clusterid=; username=; msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer - Consumer received fatal exception on startup

;org.springframework.amqp.rabbit.listener.FatalListenerStartupException: Authentication failure

        at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:367)

        at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:963)

        at java.lang.Thread.run(Thread.java:745)

Caused by: org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure

        at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:57)

        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:195)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:359)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:309)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:283)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:276)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$600(CachingConnectionFactory.java:69)

        at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:614)

        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils$1.createChannel(ConnectionFactoryUtils.java:85)

        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.doGetTransactionalResourceHolder(ConnectionFactoryUtils.java:134)

        at org.springframework.amqp.rabbit.connection.ConnectionFactoryUtils.getTransactionalResourceHolder(ConnectionFactoryUtils.java:67)

        at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.start(BlockingQueueConsumer.java:363)

        ... 2 more

Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure

        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)

        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)

        ... 12 more

Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel

        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)

        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)

        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)

        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)

        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)

        ... 15 more

2015-05-08 00:37:12,032; ERROR; servicename=; clusterid=; username=; msguuid=; nl.pharmapartners.amqp.connectors.AmqpMessageListenerContainer - Stopping container from aborted consumer

We have no real clue what happens, but it looks like a bug in spring amqp library. A new connection is started but it seems to get an already closed connection.

Can somebody explain what is happening and possibly how to fix this ?

We are running RabbitMQ 3.3.3/Erlang 17 with Spring amqp version 1.3.2, amqp-client 3.2.4.

Regards,

Wim Veldhuis.

[Gary Russell]

Please note that while the Spring team monitors this group from time to time, we might miss Spring AMQP messages here; we generally provide support on Stack Overflow (spring-amqp tag).

That said, what makes you believe this is a Spring AMQP problem; clearly from the stack trace, Spring AMQP is trying to open a connection and, in connection.start() there is some problem with a channel on that connection...

Caused by: com.rabbitmq.client.PossibleAuthenticationFailureException: Possibly caused by authentication failure

        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:373)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:516)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:545)

        at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:191)

        ... 12 more

Caused by: com.rabbitmq.client.AlreadyClosedException: clean connection shutdown; reason: Attempt to use closed channel

        at com.rabbitmq.client.impl.AMQChannel.ensureIsOpen(AMQChannel.java:190)

        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:223)

        at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:209)

        at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:202)

        at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:355)

        ... 15 more

This is entirely within the rabbitmq-client. While it is trying to authenticate, it finds the channel is closed. Take a look at the server log for more information. Notice that the exception is a PossibleAuthenticationFailureException which is an indication that the problem occurred while trying to authenticate.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Please post Spring-specific questions to StackOverflow, tagged with spring-amqp. Those projects are maintained by a different team.

Thank you.

MK

[Wim Veldhuis]

My excuses for posting in the wrong group, I will also post in the group indicated.

Just to give some additional information, the problem is the service is running for quite a while, and then suddenly stops. Other services using the same credentials running on the same system have no problem. We see this in more services we run. Restarting the service 'cures' the problem, everything works again, but it involves manual interception of the problem.'

Also the original exeption is that we are trying to use a closed channel, and we are wondering why the channel might have been closed. It feels, for us at least, that shutting down all listeners when one channel fails is a bit strange. Why shutdown everything if there is only one channel with a problem ?

 

Since posting we found that we are running a cached listener with a cachesize of one (default), prefetch of 1 and possibly 5 concurrent threads. Not a very good configuration I think, but that is a different issue.

Regards,

Wim.

[Gary Russell]

Spring AMQP changed authentication errors to be fatal after in 1.2.1, as a result of user requests (no point in keep trying if the credentials are bad).

However, it appears there is a race condition in the rabbitmq-client in that if a connection is closed right after it opens (before authentication), it is reported as a PossibleAuthenticationFailureException.

We could add an option to treat PossibleAuthenticationFailureException as non-fatal (and keep trying). Feel free to open up a JIRA issue and we'll take a look.

[Michael Klishin]

 On 9 May 2015 at 23:06:55, Gary Russell (grus...@pivotal.io) wrote:
> However, it appears there is a race condition in the rabbitmq-client
> in that if a connection is closed right after it opens (before
> authentication), it is reported as a PossibleAuthenticationFailureException.

That's not really a race condition: the protocol dictates that authentication
failures must result in a closed TCP connection. RabbitMQ has an extension that
reports them using a separate method, but that was only added in 3.x.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ