Rabbitmq different cluster name reported from each node

[Christos Roussakis]

I have a created a rabbitmq cluster with 3 nodes, if i check the cluster status with the rabbitmq command  i can see all the nodes from each machine, but each node reports a different cluster name (the hostname of each machine), i thought this is normal but i used the rabbitmq overview dashboard for grafana and i notced that it reports that i have one node per cluster, is this normal, or is it some misconfiguration on my end;

[Luke Bakken]

Hello,

We need information!

• Erlang and RabbitMQ version
• How did you form your cluster? Do you have a transcript of the commands run and their output?
• Can you share your RabbitMQ logs?

[Xiaoyan Song]

Same as me. The issue is also on 3.10.10.  But not on 3.8.3.

RabbitMQ 3.11.2 "rabbitmq_identity_info" label "rabbitmq_cluster" is different for different node. In previous version (3.7 upgraded to 3.8) it is same for all nodes.

Could you please confirm? Because this break grapha dashboard display

https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/

==============================

SLB-HGS93Z2:/mnt/c/work$ opsagent ansible -b rabbitmq -m shell -a "curl -s localhost:15692/metrics | grep identity"
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
[WARNING]: Consider using the get_url or uri module rather than running 'curl'.  If you need to use command because get_url or uri is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in
ansible.cfg to get rid of this message.
int-rmq22 | CHANGED | rc=0 >>
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@int-rmq22",rabbitmq_cluster="rab...@int-rmq22.umdptk3bwbhuhagqgg00tybkjc.dx.internal.cloudapp.net",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-Z7tt5phszIj_TuFPlDnK-g"} 1
int-rmq21 | CHANGED | rc=0 >>
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@int-rmq21",rabbitmq_cluster="rab...@int-rmq21.umdptk3bwbhuhagqgg00tybkjc.dx.internal.cloudapp.net",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-Z7tt5phszIj_TuFPlDnK-g"} 1
int-rmq20 | CHANGED | rc=0 >>
# TYPE rabbitmq_identity_info untyped
# HELP rabbitmq_identity_info RabbitMQ node & cluster identity info
rabbitmq_identity_info{rabbitmq_node="rabbit@int-rmq20",rabbitmq_cluster="rab...@int-rmq20.umdptk3bwbhuhagqgg00tybkjc.dx.internal.cloudapp.net",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-Z7tt5phszIj_TuFPlDnK-g"} 1

Reply all

Reply to author

Forward



[Luke Bakken]

Hi everyone,

Please re-read my message, where I request some information. I can't / won't provide assistance without it.

On Thursday, November 3, 2022 at 6:48:30 AM UTC-7 xs...@slb.com wrote:

Same as me. The issue is also on 3.10.10.  But not on 3.8.3.

RabbitMQ 3.11.2 "rabbitmq_identity_info" label "rabbitmq_cluster" is different for different node. In previous version (3.7 upgraded to 3.8) it is same for all nodes.

Could you please confirm? Because this break grapha dashboard display

https://grafana.com/grafana/dashboards/10991-rabbitmq-overview/

[Luke Bakken]

Hello everyone,

I am using the following project to demonstrate this issue as well as its resolution - https://github.com/lukebakken/docker-rabbitmq-cluster

Here is what I see when using RabbitMQ 3.8.3:

$ for SVC in rmq0 rmq1 rmq2; do docker compose exec "$SVC" /bin/sh -c 'curl -s localhost:15692/metrics | grep -vE '^#' | grep -F rabbitmq_identity_info'; done
rabbitmq_identity_info{rabbitmq_node="rab...@rmq0.local",rabbitmq_cluster="rabbit@rmq1"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq1.local",rabbitmq_cluster="rabbit@rmq1"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq2.local",rabbitmq_cluster="rabbit@rmq1"} 1

Notice that the cluster name is the name of the first node in the cluster.

Here is what I see using RabbitMQ 3.11.x:

$ for SVC in rmq0 rmq1 rmq2; do docker compose exec "$SVC" /bin/sh -c 'curl -s localhost:15692/metrics | grep -vE '^#' | grep -F rabbitmq_identity_info'; done
rabbitmq_identity_info{rabbitmq_node="rab...@rmq0.local",rabbitmq_cluster="rabbit@rmq0",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq1.local",rabbitmq_cluster="rabbit@rmq1",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq2.local",rabbitmq_cluster="rabbit@rmq2",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1

Notice that rabbitmq_cluster_permanent_id is consistent across all nodes. Now I will set the cluster name and re-query:

$ docker compose exec rmq0 rabbitmqctl set_cluster_name 'rabbitmq-users-A8v8XSmnCnA'
Setting cluster name to rabbitmq-users-A8v8XSmnCnA ...

$ for SVC in rmq0 rmq1 rmq2; do docker compose exec "$SVC" /bin/sh -c 'curl -s localhost:15692/metrics | grep -vE '^#' | grep -F rabbitmq_identity_info'; done
rabbitmq_identity_info{rabbitmq_node="rab...@rmq0.local",rabbitmq_cluster="rabbitmq-users-A8v8XSmnCnA",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq1.local",rabbitmq_cluster="rabbitmq-users-A8v8XSmnCnA",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1
rabbitmq_identity_info{rabbitmq_node="rab...@rmq2.local",rabbitmq_cluster="rabbitmq-users-A8v8XSmnCnA",rabbitmq_cluster_permanent_id="rabbitmq-cluster-id-1Hm4ac9xDnZeD6OjSrTg7w"} 1

This demonstrates two solutions to the issue:

• Use  rabbitmq_cluster_permanent_id in the Grafana overview dashboard. This should be the default but if not, it is a bug in our dashboard. I will check with the team.
• Use the  rabbitmqctl set_cluster_name command to set a cluster name after forming your cluster.

Thanks,

Luke

[Xiaoyan Song]

Cool! Luke! I will manual replace grafana dashboard with  rabbitmq_cluster_permanent_id. 

Thanks so much!