Prometheus RabbitMQ Exporter - how to monitor few RabbitMQ servers on Kubernetes ?

[m.przy...@biotcloud.com]

Hello everyone, i'm completely new to Prometheus using it on Kubernetes cluster. I've some question which are hard to understand so far for me. I've install kube-prometheus package via helm, and that's fine i've monitoring for my cluster etc. I've got a request to monitor RabbitMQ metrics, got few of them installed for different purposes. I've installed prometheus-rabbitmq-exporter. One thing which I don't know how to pass to make it right is how to monitor more than one endpoint while using exporter config. Do i have to deploy exporter-node for each endpoint i have in my cluster ? My config for one node of exporter looks like following, default values used:

rabbitmq:
  url: http://staging-rabbitmq.staging:15672
  user: user
  password: user
  capabilities: bert,no_sort
  include_queues: ".*"
  skip_queues: "^$"

i've enabled annotations but was is the purpose of it ? As far as i know, when this is enabled endpoint should automatically appear on target list in Prometheus, am i right with this ?

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9419"
  prometheus.io/path: "/metrics"

or do I have to add scrape configs for each endpoint like this ?

  - job_name: rabbitmq-staging
    scrape_interval: 45s
    scrape_timeout:  30s
    metrics_path: "/metrics"
    static_configs:
    - targets:
      - rabbitmq-exporter-prometheus-rabbitmq-exporter.monitoring:9419

Any comments will be appreciated to help me understand how the thing works. Best regards.

[Christian Hoffmann]

On 07/20/2018 07:15 PM, m.przy...@biotcloud.com wrote:
> Hello everyone, i'm brand new in Prometheus monitoring and didn't find

On 07/22/2018 01:25 PM, m.przy...@biotcloud.com wrote:
> Hello everyone, i'm completely new to Prometheus using it on

It seems that your message got sent twice (using minor changes).

> I've got a request to
> monitor RabbitMQ metrics, got few of them installed for different
> purposes. I've installed prometheus-rabbitmq-exporter. One thing
> which I don't know how to pass to make it right is how to monitor
> more than one endpoint while using exporter config. Do i have to
> deploy exporter-node for each endpoint i have in my cluster ?

This depends on the semantics of the exporter. While not formally
defined that way (I think?), one could put exporters in the following
categories:

(1) Single-endpoint single-tenant exporters like node_exporter -- they
need to run on each instance (e.g. each machine or node in this case)
and provide metrics only relevant for that instance.

(2) Single-endpoint multi-tenant exporters such as kube-state-metrics or
cadvisor -- there is one instance per node (for example), but it
provides metrics for multiple entities (in this case: containers).

(3) Multi-endpoint exporters such as blackbox_exporter. You usually
would run only a few of them in central places and have them collect
metrics for the given endpoints (usually passed as URL parameters).

rabbitmq_exporter [1] seems to fit into the first category, so you would
have to run one per RabbitMQ service (probably next to it).

There also seems to be another exporter called
prometheus_rabbitmq_exporter [2], but it appears to work the same.

Long story short: I believe for using the existing RabbitMQ exporters
you would have to run one exporter per RabbitMQ instance.

> i've enabled annotations but was is the purpose of it ? As far as i
> know, when this is enabled endpoint should automatically appear on
> target list in Prometheus, am i right with this ?
>
> | annotations: prometheus.io/scrape:"true" prometheus.io/port:"9419"
> prometheus.io/path:"/metrics" |

These are annotations in Kubernetes, right? (Not to be confused with
annotations in Alertmanager)

Yes, they should help with kubernetes service discovery. The docs show
how to use this feature:
https://github.com/prometheus/prometheus/blob/release-2.3/documentation/examples/prometheus-kubernetes.yml#L142

> or do I have to add scrape configs for each endpoint like this ?
>

> | -job_name:rabbitmq-staging scrape_interval:45s scrape_timeout: 30s

> metrics_path:"/metrics" static_configs: -targets:
> -rabbitmq-exporter-prometheus-rabbitmq-exporter.monitoring:9419 |

That should work, too. However in dynamic environments such as
Kubernetes it might make more sense to use the kubernetes discovery.


Kind regards,
Christian

[Matthias Rampke]

As Christian said, you need one exporter process per RabbitMQ process. In Kubernetes, the way is to run them side by side in a pod (this is what pods are designed for). That way, the the exporter can connect to RabbitMQ on localhost. 

Then, set prometheus.io/port to the exporter port, and use the Kubernetes service discovery to find them all. There is an example config in the documentation. Note that the annotations are not magic – the example prometheus-kubernetes config contains target relabeling rules that use them.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fb947d00-3abf-fed1-2c73-ecdf7775d451%40hoffmann-christian.info.
For more options, visit https://groups.google.com/d/optout.