does config file get cached?

68 views
Skip to first unread message

VC

unread,
Sep 26, 2022, 10:32:46 AMSep 26
to rabbitmq-users
Hi team,
I need help in understanding whether config file gets cached anywhere or not? Is there a way to flush out the config file values?
The reason that I am asking this query is, while trying the manual Peer Discovery via the config file, we are facing challenges if we comment out the configurations. After resetting the nodes, they are getting added automatically to their prior cluster sets as if they read it from the cached config file.
Kindly help

Luke Bakken

unread,
Sep 26, 2022, 1:27:01 PMSep 26
to rabbitmq-users
Hello,

My guess is that you're not actually resetting the nodes, or you aren't commenting out the configuration correctly.

How are you resetting the nodes?
Luke

VC

unread,
Sep 27, 2022, 1:44:04 AMSep 27
to rabbitmq-users
Hi Luke,

I am commenting the config file fully on both nodes, using the # keyword.

#loopback_users.guest = false
#listeners.tcp.default = 5672
#cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config
#cluster_formation.classic_config.nodes.1 = rabbit@node-01
#cluster_formation.classic_config.nodes.2 = rabbit@node-02

Post that, these are the commands I am executing:
Stopping node-01 
rabbitmqctl stop_app

Resetting node-01
rabbitmqctl reset

Removing node-01 from cluster via node-02
rabbitmqctl forget_cluster_node rabbit@node-01

Starting node-01
rabbitmqctl start_app

Thank you

Luke Bakken

unread,
Sep 27, 2022, 10:07:46 AMSep 27
to rabbitmq-users
Thanks, I will try to reproduce this.

You're only using two nodes for your test, correct? In production an odd number of nodes is required (3 or 5 is our recommendation).

I'll give this a try using RabbitMQ 3.11.0. What version are you using?

Luke

VC

unread,
Sep 28, 2022, 12:37:44 AMSep 28
to rabbitmq-users
Yes Luke, my cluster is of 3 nodes, but I mentioned only 2 for the sake of brevity.
System:
Ubuntu 22.04
Erlang 24
Rabbitmq 3.9.13

Luke Bakken

unread,
Sep 28, 2022, 11:46:13 PMSep 28
to rabbitmq-users
That's a very important detail to omit.

Did you run  forget_cluster_node on both remaining nodes?

VC

unread,
Sep 28, 2022, 11:51:59 PMSep 28
to rabbitmq-users
Hi Luke, I simply wanted to remove one node from the cluster, hence forget_cluster_node only for a single node

Luke Bakken

unread,
Sep 29, 2022, 9:59:57 AMSep 29
to rabbitmq-users
You must run  run  forget_cluster_node on all remaining nodes (https://www.rabbitmq.com/clustering.html#removing-nodes)

VC

unread,
Oct 3, 2022, 12:05:27 AMOct 3
to rabbitmq-users
Hi Luke,
Thank you for the reply. Although the end result is still the same. The node-03 gets added back to the cluster after the steps

I ran forget_cluster_node on node-01
rmqquery2.png

After than I ran the forget_cluster_node on node-02 and this was the message I received.
rmqquery.png

Luke Bakken

unread,
Oct 4, 2022, 4:25:06 PMOct 4
to rabbitmq-users
Hello again -

Yep, I can reproduce what you're reporting. I will come up with a workaround.

I'm using this repository to keep track of my Docker configuration - https://github.com/lukebakken/rabbitmq-users-yr--HAlasOE

Thanks,
Luke

Luke Bakken

unread,
Oct 5, 2022, 11:16:31 AMOct 5
to rabbitmq-users
Hello again!

Please see this script that shows what commands are necessary to run to forget a cluster node when peer discovery is used -

Notice that, basically, every node must be restarted and that stop_app/start_app is not sufficient. So yes, as you suspected, the configuration is not re-read. 

I have opened the following issue if you'd like to follow along - https://github.com/rabbitmq/rabbitmq-server/issues/6021

It is rare to forget cluster nodes, especially when peer discovery is involved, so restarting nodes may, in fact, be the correct way to deal with this.

Thanks,
Luke

Reply all
Reply to author
Forward
0 new messages