Discarding message error in an old incarnation (3) of this node (1) although HA is disabled

[Pavel Arakelyan]

Hello,

I'm encountering problem related to "old incarnation". We don't have high availability enabled and we run single node in kubernetes, but still the rabbitmq node crushes from time to time for unknown reasons and it can't start up. There are tons of logs like this....

[error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)

RabbitMq version is 3.8.3.

Erlang version is 22.3.4.1.

Please let me know what may be the problem.

Thanks in advance.

[Wesley Peng]

Pavel Arakelyan wrote:
> I'm encountering problem related to "old incarnation". We don't have
> high availability enabled and we run single node in kubernetes, but
> still the rabbitmq node crushes from time to time for unknown reasons
> and it can't start up. There are tons of logs like this....
>
> [error] emulator Discardingmessage
> {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}}
> from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)

Seems like IO issue causes this problem.
You may check the container's IO stability.

Regards.

[Pavel Arakelyan]

We also persist queue messages, may the huge I/O result in that kind of exceptions or you mean there may be a network inconsistency?

Thanks.

[Luke Bakken]

Hi Pavel,

Those messages can be ignored during startup.

Are you monitoring your RabbitMQ node? Can you archive and provide your complete log files?

Thanks -

Luke

[Pavel Arakelyan]

Hello Luke,

You mean there is an option to ignore the messages on which the mentioned error happens during startup? If yes then how?

Yes we do monitor the RabbitMQ node. The thing is that error does not give a hint what may have gone wrong, what does "incarnation (3) of this node (1)" actually mean?

I'll try to send you logs, right now I can send you the RabbitMQ config that we use, you may have some comments on it.

Thanks,

Regards.

[Luke Bakken]

Hi Pavel,

There is no way to suppress the messages. You may see them on restarts of your node and they generally do not indicate an error.

If you are only running one node there is no need to use any peer discovery backend.

There is still not enough information to assist. We don't know why RabbitMQ is crashing nor are there any details about why "it can't start up".

Thanks,

Luke

[Pavel Arakelyan]

Hello Luke,

I have attached the logs of some period when restarts begun and it was restarting over and over again until I remove folder under vhosts. And as I stated before this happens from time to time.

Hope the logs help.

Thanks,

Regards.