Investing the cause of "Missing 'DOWN' message"

[JK]

Hello fellow users,

We had an issue where client connections were timing out. When we looked into the cause we saw these messages in the logs:

2022-05-07 20:39:02.698 [warning] <0.25758.56> Mirrored queue 'item-sync.process-item-bucket-6.delay.1000' in vhost 'item-sync': Missing 'DOWN' message from <21957.1398.4594> in node 'rabbit@rmq-item-sync1-2a'
2022-05-07 20:39:04.236 [warning] <0.32465.56> Mirrored queue 'item-sync.process-item-bucket-5.delay.1000' in vhost 'item-sync': Missing 'DOWN' message from <21957.23285.7862> in node 'rabbit@rmq-item-sync1-2a
'2022-05-07 20:39:04.352 [warning] <0.19240.58> Mirrored queue 'item-sync.process-item-bucket-4.delay.1000' in vhost 'item-sync': Missing 'DOWN' message from <21957.27477.3298> in node 'rabbit@rmq-item-sync1-2a'
2022-05-07 20:39:05.772 [warning] <0.2834.58> Mirrored queue 'item-sync.process-item-bucket-7.delay.1000' in vhost 'item-sync': Missing 'DOWN' message from <21957.27381.4511> in node 'rabbit@rmq-item-sync1-2a' 

This was from a node named rmq-item-sync1-4a, complaining about the remote node. We resolved it by restarting the problem node (2a) and haven't seen it come back yet. We would still like to figure out the cause so we can prevent this in the future, but I couldn't find anything definitive when searching for this error message.

Thanks!

[Yong Walt]

what's the software version?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ef0a3ee0-fdec-401c-86a6-bbc246416688n%40googlegroups.com.

[Michal Kuratczyk]

Hi,

This message is logged when a mirror doesn't report it stopped when it was asked to:

https://github.com/rabbitmq/rabbitmq-server/blob/master/deps/rabbit/src/rabbit_mirror_queue_misc.erl#L345

As the comment suggests, it can be a network partition or some other connectivity issue where the mirror didn't stop as expected or stopped but didn't report back.

Ultimately the solution is to move to quorum queues. I believe in the past the lack of TTL support in quorum queues was a problem for you. RabbitMQ 3.10 added TTL+DLX support in quorum queues.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAPW%3DFH2_FrWvjP%3DiDUJydCuDttqyJJe4g5akqVvVqTw2vCs2YA%40mail.gmail.com.

--

Michał

RabbitMQ team