Discading message error during RabbitMQ launching

[Yusuke Masuda]

Hi.

I found the log messages below,

2019-10-08 17:46:25.541 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)
2019-10-08 17:46:25.541 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.442.0> in an old incarnation (3) of this node (1)

I wonder those are saying something wrong.

As I saw the thread below, they might not be error, just normal behavior, they are not strictly same though.

Also our environment is not configured cluster, just standalone.

https://groups.google.com/forum/#!msg/rabbitmq-users/QNTkoKrg2H0/84OJZ7juEgAJ

Please let me know it is normal, or not.

Sorry to ask such a basic question.

Regards,

Yusuke

[Wesley Peng]

Hello,

on 2019/10/9 18:55, Yusuke Masuda wrote:
> |
> 2019-10-0817:46:25.541[error]emulator Discardingmessage

> {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}}
> from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)
> 2019-10-0817:46:25.541[error]emulator Discardingmessage
> {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}}
> from <0.342.0> to <0.442.0> in an old incarnation (3) of this node (1)
> |
>

Before the community can help you, we must know this information:

Version of Erlang and RabbitMQ
Operating system and version
RabbitMQ configuration files
RabbitMQ log files, or log file entries
Exact error output
Exact commands you are running, or code you are running

regards.

[Michael Klishin]

The message means that a restarted queue [mirror] process has received commands directed at its

earlier "incarnation" (its version before the restart). They are discarded as the queue has a different master now.

[1][2][3] are relevant doc guides.

This should be a warning, not an error, really.

1. https://www.rabbitmq.com/ha.html#behaviour

2. https://www.rabbitmq.com/ha.html#unsynchronised-mirrors

3. https://www.rabbitmq.com/quorum-queues.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/56b84cce-737b-4f21-8b42-753e3e34d5bd%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Pavel Arakelyan]

Hello Michael,

I'm encountering exactly this problem related to "old incarnation". We don't have high availability enabled and we run single node in kubernetes, but still the rabbitmq node crushes from time to time for unknown reasons and it can't start up. There are tons of logs like this....

[error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)

RabbitMq version is 3.8.3.

Please let me know what may be the problem or where can we discuss this in detail.

Thanks in advance.

On Wednesday, October 9, 2019 at 7:13:20 PM UTC+4, Michael Klishin wrote:

The message means that a restarted queue [mirror] process has received commands directed at its

earlier "incarnation" (its version before the restart). They are discarded as the queue has a different master now.

[1][2][3] are relevant doc guides.

This should be a warning, not an error, really.

1. https://www.rabbitmq.com/ha.html#behaviour

2. https://www.rabbitmq.com/ha.html#unsynchronised-mirrors

3. https://www.rabbitmq.com/quorum-queues.html

On Wed, Oct 9, 2019 at 5:55 AM Yusuke Masuda <yusuke...@gmail.com> wrote:

Hi.

I found the log messages below,

2019-10-08 17:46:25.541 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.439.0> in an old incarnation (3) of this node (1)
2019-10-08 17:46:25.541 [error] emulator Discarding message {'$gen_cast',{force_event_refresh,#Ref<0.211810718.604766209.11182>}} from <0.342.0> to <0.442.0> in an old incarnation (3) of this node (1)

I wonder those are saying something wrong.

As I saw the thread below, they might not be error, just normal behavior, they are not strictly same though.

Also our environment is not configured cluster, just standalone.

https://groups.google.com/forum/#!msg/rabbitmq-users/QNTkoKrg2H0/84OJZ7juEgAJ

Please let me know it is normal, or not.

Sorry to ask such a basic question.

Regards,

Yusuke

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/56b84cce-737b-4f21-8b42-753e3e34d5bd%40googlegroups.com.

[Deshdeepak Shukla]

We faced the same issue in one of our three node 3.8.3.cluster. one of the node was spitting this message continuously and lead to certain queues being not created. 

The following list of steps helped remediate this issue
1. Node with the continuous log (emulator Discarding message) was temporarily remediated by "rabbitmqctl stop_app; rabbitmqctl start_app"

2. in order to fully remediate the issue, the entire cluster was restarted in an ordered manner (two ram nodes were stopped first followed by disk)  and the startup was performed in reversed order (disk node first and then the ram nodes)

After performing step 2 the cluster fully recovered . 

could it be caused by the following bug?
https://github.com/rabbitmq/rabbitmq-server/commit/3c1b317b08870d3ca4c04c4a59b4067bf768707e
 
Cheers!

Desh Shukla