Can't join cluster after upgrade, even with same feature flags enabled

[João Olavo Vasconcelos]

Dears,

node1 was on version 3.8.9 and was upgraded to 3.8.21.

node2 was already on version 3.8.21.

I'm trying to join node2 to node1, but I'm getting the incompatible_feature_flags error.

# on node2

rabbitmqctl join_cluster rabbit@node1
Clustering node rabbit@node2 with rabbit@node1
Error:
incompatible_feature_flags

They both have the same feature flags.

node1 had the drop_unroutable_metric, empty_basic_get_metric and user_limits disabled, and I enabled them. node2 had already all feature flags enabled.

rabbitmqctl list_feature_flags
Listing feature flags ...
name    state
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
implicit_default_bindings       enabled
maintenance_mode_status enabled
quorum_queue    enabled
user_limits     enabled
virtual_host_metadata   enabled

What I saw as different is that node1 has file mnesia/rabbit@node1-feature_flags with the content:

[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 maintenance_mode_status,quorum_queue,user_limits,virtual_host_metadata].

And node2 has the respective file mnesia/rabbit@node2-feature_flags with the content:

[drop_unroutable_metric,empty_basic_get_metric,implicit_default_bindings,
 quorum_queue,user_limits,virtual_host_metadata].

That is, node2 doesn't explicit the maintenance_mode_status flag.

What am I missing? How to join them on a cluster?

Thanks in advance!

João Olavo Vasconcelos

[Michal Kuratczyk]

Hi,

The need to have the same feature flags enabled, not just listed. So it sounds like in your case you need to do "rabbitmqctl enable_feature_flag maintenance_mode_status"

More info https://www.rabbitmq.com/feature-flags.html#how-to-enable-feature-flags.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/cfeb21d2-8fb1-49cf-87ef-f5837476a959n%40googlegroups.com.

--

Michał

RabbitMQ team

[João Olavo Vasconcelos]

Michael, thanks for the response.

Sorry, I wasn't explicit on my first message: all flags are already enabled.

The difference I pointed out on maintenance_mode_status was only on a file. Consulting "rabbitmqctl list_feature_flags", it shows that, on both nodes, all flags are enabled, including maintenance_mode_status, and they both have the same flags.

On node1:

# rabbitmqctl list_feature_flags

Listing feature flags ...
name    state
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
implicit_default_bindings       enabled
maintenance_mode_status enabled
quorum_queue    enabled
user_limits     enabled
virtual_host_metadata   enabled

On node2:

# rabbitmqctl list_feature_flags

Listing feature flags ...
name    state
drop_unroutable_metric  enabled
empty_basic_get_metric  enabled
implicit_default_bindings       enabled
maintenance_mode_status enabled
quorum_queue    enabled
user_limits     enabled
virtual_host_metadata   enabled

What am I missing?

[Michal Kuratczyk]

I'm not sure, please share full logs from all nodes when this happens.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/994ab0e1-7181-46c1-b00a-510d601f7b64n%40googlegroups.com.

--

Michał

RabbitMQ team

[João Olavo Vasconcelos]

I figured out what happened.

There was a "ghost" process running from the last version/container. docker stop and docker rm didn't make the complete stop and another rabbitmq-server process was running on server and receiving the connections.

So, the plugins weren't enabled on that other old process.

Restart docker daemon (and make sure that that process was killed) solved it.