Discarding message errors leading to RabbitMQ instability

[Ajmal Rahman]

We are seeing issues with our RabbitMQ on production, as we see problems in publishing messages to exchanges\queues. We se many errors like below in the logs.

2021-06-17 08:29:21.002 [error] emulator Discarding message {'$gen_call',{<0.1172.0>,#Ref<0.2127539016.512753668.230792>},{info,[state]}} from <0.1172.0> to <0.22586.6> in an old incarnation (1623843588) of this node (1623894858) 

When I checked different posts regarding the same issue, I read that it is to be expected during an upgrade\restart. But we see the errors even after a few hours of it running.

We had upgraded quite recently to the latest version(s), in the expectation that this issue will get fixed (as it was being thrown in the old version we had). But it did not help. The versions are as below:

RabbitMQ 3.8.17

Erlang 23.3.4.2

We are running it as HA - with 3 nodes running. Further good to knows:

• We are seeing this only on one of our production environments (out of 6) - with the same configuration. Also others handle more messages and thereby has more load.
• We have setup an ha-mirroring policy for all the queues (ha-mode: exactly / count 2/ automatic sync mode)
• We do not see this error in other environment(s), which are stable.

We could only co-relate this issue we are facing with this error message - rest of the stats in the Overview seems to be fine. And when it runs stable for a while, these errors are not there in the logs.

What could be causing this issue and how could we fix it? Any suggestions \ ideas what to look for \ improve?

Regards

Ajmal

[xianming ma]

Hi, Ajmal:

    I don't know how to solve it. The community suggests replacing the classic mirrored queues with the  quorum queues. I haven't tried yet.

Regards

Carl

Ajmal Rahman <ara...@een.com> 于2021年6月17日周四 下午5:02写道：

Smart Video Surveillance - #1 in Cloud Video Surveillance Worldwide

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/5930e13f-204a-465d-959b-f192a3d196cbn%40googlegroups.com.