autorecovery failure on queue status operation

[Dave Tenny]

I have a number of rabbitmq clients, most of which fail over well enough, but one client is experiencing apparent thread death on a queue status operation when the connection times out on heartbeat.

I wondered if anybody had any clues here.  Note the langohr.queue$status.invoke()

Exception in thread "async-dispatch-3" java.lang.Error: java.io.IOException

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1151)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException

at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:106)

at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:102)

at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:124)

at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:873)

at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.queueDeclarePassive(AutorecoveringChannel.java:287)

at langohr.queue$status.invoke(queue.clj:130)

at de.data.messaging.scheduler$fill_jobs_BANG_.invoke(scheduler.clj:115)

at de.data.messaging.scheduler$go_release_jobs_BANG_$fn__1064$state_machine__11049__auto____1065$fn__1067.invoke(scheduler.clj:129)

at de.data.messaging.scheduler$go_release_jobs_BANG_$fn__1064$state_machine__11049__auto____1065.invoke(scheduler.clj:129)

at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:940)

at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:944)

at clojure.core.async.impl.ioc_macros$take_BANG_$fn__11065.invoke(ioc_macros.clj:953)

at clojure.core.async.impl.channels.ManyToManyChannel$fn__7516.invoke(channels.clj:102)

at clojure.lang.AFn.run(AFn.java:22)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

... 2 more

Caused by: com.rabbitmq.client.ShutdownSignalException: connection error

at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:67)

at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:33)

at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:348)

at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:221)

at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:118)

... 14 more

Caused by: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds

at com.rabbitmq.client.impl.AMQConnection.handleSocketTimeout(AMQConnection.java:597)

at com.rabbitmq.client.impl.AMQConnection.access$600(AMQConnection.java:65)

at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:560)

... 1 more

[Dave Tenny]

Langohr 3.2.0

[Michael Klishin]

 On 24 Jul 2015 at 21:21:39, Dave Tenny (dave....@gmail.com) wrote:
> I have a number of rabbitmq clients, most of which fail over well
> enough, but one client is experiencing apparent thread death
> on a queue status operation when the connection times out on heartbeat.

What do you mean by “thread death”? There’s simply a timeout exception
in a thread pool where consumer functions are executed.
--
@michaelklishin, github.com/michaelklishin

[Dave Tenny]

The thread on which that event occurs becomes unresponsive.  The queue status call is happening on a clojure.core.async thread servicing <! actions, but stops responding after that exception despite more posts to the channel to which <! is listening.

[Dave Tenny]

Of course it's possible it has nothing to do with Langohr/RabbitMQ, but most of our code works just fine with the Langohr autorecovery stuff, i.e. we don't have to do much, if anything.  It's only this thread doing the queue status operation that is going AWOL.

[Michael Klishin]

DefaultExceptionHandler in Java closes the channel after an unhandled exception. There is more forgiving version available in 3.5.4. Doing your own handling is a good idea anyway.

MK