What could be the reason rabbit queue stays in down state and can't recover to idle state?

[lang qiu]

When I restart rabbit mq server the queues normally will go through the down state to the idle state quickly but recently we hit a situation that many queues stay in the down state. So we have many logs show something like

Server channel error: 404, message: NOT_FOUND - home node 'rabbit@RabbitMQ_02' of durable queue 'callback_queue_5f27b7f3628d8' in vhost '/' is down or inaccessible /var/www/html/app/Services/RabbitMQ.php185

What can the reason for that?

Thanks,

Qiulang

[jo...@cloudamqp.com]

Hi,

There is not that much information to work with here such as RabbitMQ and Erlang versions, complete logs - especially what happened right before the first "down"-message. This usually reveals the reason for the "down"-state.

You can try "rabbitmq-diagnostics list_unresponsive_queues" to see which queues RabbitMQ consider to be down.

You might eventually try "rabbitmqctl eval '{_,Q} = rabbit_amqqueue:lookup(rabbit_misc:r(<<"/">>, queue, <<"callback_queue_5f27b7f3628d8">>)), erlang:exit(rabbit_amqqueue:pid_of(Q), kill).'" to forcefully restart the queue(s). But first find out why it is down.

/Johan

[lang qiu]

Hi thanks for looking into my question.  We are using RabbitMQ 3.7.5 Erlang 20.3. I did check the log after restart the mq server but it didn't provide any clue. 

The log just said "Socket error: could not connect to host" then I got 404 error. Please checked the attached log images

[jo...@cloudamqp.com]

No, we need the server logs at that time. But since you are running RabbitMQ 3.7 there is not much we can do. Update to RabbitMQ 3.12.x or 3.13.1.

/Johan