Migrating Monitoring from Management Plugin to Prometheus Plugin

[Ilia Kurenkov]

I tried to draw parallels between the metrics coming from the Management Plugin and the newer Prometheus Plugin.

Just based on the available documentation for the metrics, it seems there's a significant difference in the kinds of metrics the two plugins expose. Is this true? I'm not versed in Erlang, so I felt lost trying to find the answer in the source code myself.

Thanks for your help!

[Ilia Kurenkov]

As a related follow-up:

It seems that the management plugin endpoint exposes metrics about all nodes in the cluster, whereas the prometheus endpoint will only report objects from the node its on, with the exception of things like quorum queues.

If I have a cluster of many nodes and I want to monitor them with the Prometheus plugin, what's a good way to automatically discover these nodes? I wouldn't want to manually have to add endpoints for each one.

[Michal Kuratczyk]

Hi,

Yes, there is an overlap in the metrics but there are also different metrics exposed only by one of the endpoints. If there is anything available through the Management API that you wish was available through the Prometheus endpoint, that'd be valuable feedback.

And yes, you get metrics from a single node when you query the Prometheus endpoint. This is one of the most important distinctions and improvements. In a large system, with many queues and connections, the Management API can be very slow to respond as it needs to query all details from all nodes. Prometheus' endpoint responds with its local data only. Prometheus (and other monitoring systems that support Prometheus endpoints/format) have service discovery mechanisms:

https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d840bf94-fa3c-4b68-82fe-27388cbea255n%40googlegroups.com.

--

Michał

RabbitMQ team

[Ilia Kurenkov]

Thanks Michal for the response!

Regarding the metrics overlap, could you point me to some leads from which I can assemble a list of matches/mismatches. I have created a list based on comparing descriptions, but that experience makes me think either some docs by the devs or even source code would be a better source of truth.

[Michal Kuratczyk]

We don't have such a list. Many of the metrics are collected in a different way so it's not a simple change of metric name that you can find in the source code.

We published dashboards that should get you started with Prometheus-based monitoring. https://grafana.com/orgs/rabbitmq/dashboards

There are also a few markdown files with lists of metrics per endpoint (we actually have a few Prometheus endpoints):

https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_prometheus

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d9001ec9-20d7-43e5-80e5-d265d12c2de7n%40googlegroups.com.

--

Michał

RabbitMQ team

[Matt Green]

I too found the process of working out where to get stats from and how a little confusing. However I think I've got the hang of it now and the stats I was able to get from the new "detailed" endpoint are much quicker than using the Management UI. I don't know if it helps but I use "rabbitmqctl cluster_status" to get a view of which nodes are in the cluster and if there are any problems, like a node down or a split brain.

I then use the list of nodes from there to know where I have to go and look for Prometheus stats, that gets uploaded to a DB where I can run reports or issue alerts.

As I say, I'm very pleased with how much faster this is than having to scrape the Management UI. If I were to wish for anything at the moment it would be for a list of current connections open by users. Getting it from the Management UI is either slow or it just hangs (at least for me) so if that were available from Prometheus it would help.

Cheers,

Matthew.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0vbg_hQMM-xPry6SuD-dY3XjmXqy9xNnG5K35%3DJ%2Bz4EQg%40mail.gmail.com.

[Michal Kuratczyk]

Do you mean a count of connections per user or more than that?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAMLWFk82EXg4-1sd4TajvVEQ6bDMnBLyV6tkDDd3HKVf8yNzzw%40mail.gmail.com.

--

Michał

RabbitMQ team

[Matt Green]

I mean exactly that :-) I tried using "rabbitmqctl list_connections user" but it seems to hang (I would have then had to process the output to get a count per user).

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA81d0t%3DZOhUcBDs%3DSvhBkttZhCw1az8sCc4x4b1snz2_Ye-1w%40mail.gmail.com.

[Ilia Kurenkov]

Oh, Matt, interesting regarding the cluster info!

I'm particularly curious if that data can be used for auto-discovery of the nodes :)

[Michal Kuratczyk]

Prometheus has multiple service discovery mechanisms. Why are you looking for something different?

As for connections per user, I guess it could be added. We already track this information (because you can set the maximum number of connections per user),

so it's just a matter of exposing it.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/c0cbe8fc-3774-42f0-a1bb-64f0fde2d432n%40googlegroups.com.

--

Michał

RabbitMQ team

[Ilia Kurenkov]

We're getting rabbitmq to talk to Datadog, so we can't really take advantage of Prometheus autodiscovery.