Message published to topic exchange is not delivered to all subscribing queues

[Franciszek Bogacki]

Hi, 

we are currently running RabbitMQ 3 nodes cluster (versions RMQ: 3.8.30, Erland: 23.3.4.11) and we've spotted a very strange behaviour during e2e tests. Messages put to the topic exchange stopped being delivered to selected queues. Naturally we looked at the configuration in the UI, all queues were declared the same way with the same bounding key. 

Then we stopped all the consuming microservices and put the message manually to the exchange, same behaviour. 

We searched the rabbitmq logs and found no errors, only that one node have failed the night before but was successfully brought up:

 

2022-09-26 23:45:13.693 [warning] <0.25764.7> Received a 'DOWN' message from rabbit@yb8162 but still can communicate with it 2022-09-26 23:45:20.827 [info] <0.590.0> rabbit on node rabbit@yb8163 down 2022-09-26 23:45:20.833 [info] <0.590.0> Keeping rabbit@yb8163 listeners: the node is already back 2022-09-26 23:45:20.902 [info] <0.590.0> 8 transient queues from an old incarnation of node rabbit@yb8163 deleted in 0.068931s 2022-09-26 23:45:24.279 [info] <0.590.0> rabbit on node rabbit@yb8163 up

What do we miss here and how to prevent it happening in the future? Is it a known bug and is fixed in the newer vesions?

[Michal Kuratczyk]

Hi,

3.8 is no longer supported so you should upgrade anyway

You can use https://www.rabbitmq.com/firehose.html to trace messages (the trace will include a "routed queues" field and may give you an idea of what's going on).

If you still observe this in a recent RabbitMQ version and believe it's a bug, we would need a reproducible test case - even if it doesn't always happen / doesn't happen in every environment,

the exact steps you perform to declare the exchange/queue/bindings and to publish and consume a message, may shed some light on this.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e7d68b7b-8427-46e6-bdc5-96ccf92ad703n%40googlegroups.com.

--

Michał

RabbitMQ team

[Franciszek Bogacki]

Thank you for the reply, we are upgrading to the newer version as we speak. Not sure which exactly which but for sure on RHEL8 this time.

I will also look into enabling firehose, thank you :)

[Franciszek Bogacki]

After upgrade, we have now RabbitMQ 3.10.7Erlang 25.0.3. I hope it is supported :)

So far no issues but we did not have much time to test it, yet.