does config file get cached?

[VC]

Hi team,

I need help in understanding whether config file gets cached anywhere or not? Is there a way to flush out the config file values?

The reason that I am asking this query is, while trying the manual Peer Discovery via the config file, we are facing challenges if we comment out the configurations. After resetting the nodes, they are getting added automatically to their prior cluster sets as if they read it from the cached config file.

Kindly help

[Luke Bakken]

Hello,

My guess is that you're not actually resetting the nodes, or you aren't commenting out the configuration correctly.

How are you resetting the nodes?

Luke

[VC]

Hi Luke,

I am commenting the config file fully on both nodes, using the # keyword.

#loopback_users.guest = false
#listeners.tcp.default = 5672
#cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
#cluster_formation.classic_config.nodes.1 = rabbit@node-01
#cluster_formation.classic_config.nodes.2 = rabbit@node-02

Post that, these are the commands I am executing:

Stopping node-01 

rabbitmqctl stop_app

Resetting node-01

rabbitmqctl reset

Removing node-01 from cluster via node-02

rabbitmqctl forget_cluster_node rabbit@node-01

Starting node-01

rabbitmqctl start_app

Thank you

[Luke Bakken]

Thanks, I will try to reproduce this.

You're only using two nodes for your test, correct? In production an odd number of nodes is required (3 or 5 is our recommendation).

I'll give this a try using RabbitMQ 3.11.0. What version are you using?

Luke

[VC]

Yes Luke, my cluster is of 3 nodes, but I mentioned only 2 for the sake of brevity.

System:

Ubuntu 22.04

Erlang 24

Rabbitmq 3.9.13

[Luke Bakken]

That's a very important detail to omit.

Did you run  forget_cluster_node on both remaining nodes?

[VC]

Hi Luke, I simply wanted to remove one node from the cluster, hence forget_cluster_node only for a single node

[Luke Bakken]

You must run  run  forget_cluster_node on all remaining nodes (https://www.rabbitmq.com/clustering.html#removing-nodes)

[VC]

Hi Luke,

Thank you for the reply. Although the end result is still the same. The node-03 gets added back to the cluster after the steps

I ran forget_cluster_node on node-01

After than I ran the forget_cluster_node on node-02 and this was the message I received.

[Luke Bakken]

Hello again -

Yep, I can reproduce what you're reporting. I will come up with a workaround.

I'm using this repository to keep track of my Docker configuration - https://github.com/lukebakken/rabbitmq-users-yr--HAlasOE

Thanks,

Luke

[Luke Bakken]

Hello again!

Please see this script that shows what commands are necessary to run to forget a cluster node when peer discovery is used -

https://github.com/lukebakken/rabbitmq-users-yr--HAlasOE/blob/main/remove-rmq0.sh

Notice that, basically, every node must be restarted and that stop_app/start_app is not sufficient. So yes, as you suspected, the configuration is not re-read. 

I have opened the following issue if you'd like to follow along - https://github.com/rabbitmq/rabbitmq-server/issues/6021

It is rare to forget cluster nodes, especially when peer discovery is involved, so restarting nodes may, in fact, be the correct way to deal with this.

Thanks,

Luke