Grafana/Prometheus - Metric for rebalance alert

[Aldo Bongio]

Hi, 

  with version 3.8 I wonder if there is a metric or a combination of Prometheus metrics that permits to build a Grafana alert indicating that a queue rebalance is needed.

For instance if I execute rabbitmq-queues rebalance all right now on my cluster of 3 nodes I obtain the following output, and nothing change:

Rebalancing all queues...

┌────────────────────┬─────────────────────────┬──────────────────────────┐

│ Node name          │ Number of quorum queues │ Number of classic queues │

├────────────────────┼─────────────────────────┼──────────────────────────┤

│ rabbitmq@rabbitmq6 │ 0                       │ 15                       │

├────────────────────┼─────────────────────────┼──────────────────────────┤

│ rabbitmq@rabbitmq5 │ 0                       │ 15                       │

├────────────────────┼─────────────────────────┼──────────────────────────┤

│ rabbitmq@rabbitmq4 │ 0                       │ 27                       │

└────────────────────┴─────────────────────────┴──────────────────────────┘

Moreover, if I get the rabbitmq_queues Prometheus metric right now for the cluster I obtain:

new_rabbitmq_queues{instance="rabbitmq4"} 51

new_rabbitmq_queues{instance="rabbitmq5"} 15

new_rabbitmq_queues{instance="rabbitmq6"} 15

The difference (51 vs 27) on node 4 is because such node contains 24 springCloudBus.anonymous.* non-mirrored queues.

In the end, my questions are:

1. Is there a way to obtain 27 instead of 51 from Prometheus for node 4, thus filtering out the non mirrored queues ?
2. Is there a simple formula such as max(...) - min(...) > SOME_VALUE indicating that issuing a rabbitmq-queues rebalance all will trigger a real rebalance ? As I stated before, with the current state (15, 15, 27) the cluster is fine and trying a rebalance results in a no-op.

Thanks

[Gerhard Lazu]

Hi,

1. Is there a way to obtain 27 instead of 51 from Prometheus for node 4, thus filtering out the non mirrored queues ?

It does make sense to categorise queues by type, especially since we intend to introduce more queue types. Even though this is currently not possible, it's a legitimately good idea, thanks! https://github.com/rabbitmq/rabbitmq-prometheus/issues/16

 Is there a simple formula such as max(...) - min(...) > SOME_VALUE indicating that issuing a rabbitmq-queues rebalance all will trigger a real rebalance ? As I stated before, with the current state (15, 15, 27) the cluster is fine and trying a rebalance results in a no-op.

Rebalance only works for replicated queues (currently mirrored or quorum): https://github.com/rabbitmq/rabbitmq-server/blob/7138c748b317a23cf103687cf03d57aed8feb400/src/rabbit_amqqueue.erl#L501

Without categorising queues by type - as captured in your issue - we cannot indicate whether a rebalance would help.