Cannot get scraping of Prometheus metrics to work

[Scott Mackie]

I'm trying to set up monitoring of RabbitMQ using Prometheus in a Kubernetes cluster.

I've been following the guide on how to set this up using both the Prometheus and RabbitMQ operators for Kubernetes. However when I deploy the PodMonitor (for rabbitmq-operator) and ServiceMonitors (for the rabbitMQ cluster itself) to the cluster, Prometheus doesn't seem to be scraping the metrics as expected.

I've correctly set the metadata.labels.release property in the YAML for these two resources to match what the Prometheus operator is expecting, and can see them listed in Prometheus' Status -> Service Discovery UI, but the 'active targets' always reports 0.

My current suspicion is that there are no prometheus or prometheus-tls ports declared against the rabbitMQ Service in the cluster, which is what the ServiceMonitor is expecting to scrape from. Presumably declaring this port on the Service is controlled by the rabbitMQ cluster operator. The documentation doesn't mention any additional steps to set up these ports, so I'm not sure if I am understanding the problem correctly.

[Michal Kuratczyk]

Hi,

As long as Prometheus can't find the target, it's more of a Prometheus problem/question than a RabbitMQ question.

Of course, I would love our docs to be more reliable so if you find what's wrong - by all means tell us (ideally by a PR to the docs),

but Prometheus Operator troubleshooting guide is probably the right place to start:

https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/troubleshooting.md#troubleshooting-servicemonitor-changes

The ports are exposed by default, so this should not be the problem.

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ab5fc8ac-9dc8-4ad5-991b-6afb10dae202n%40googlegroups.com.

--

Michał

RabbitMQ team

[Scott Mackie]

I ended up getting this working, at least partially. It appears that my suspicion was correct. I had to override the RabbitMQCluster resource to expose the prometheus service in addition to the other ports that the Service was already exposing. I did this by following the additionalPorts example from the GitHub repo for the cluster operator.

override:
  service:
    spec:
      ports:
        - name: prometheus
        protocol: TCP
        port: 15692
        targetPort: 15692

As soon as I did this, Prometheus was able to see the targets. It's worth pointing out though that not all of the targets worked correctly. The targets for /metrics worked, but a second set of targets for /metrics/detailed did not work. I saw 404 errors for those targets. The ServiceMonitor suggested by the documentation includes both of these endpoints, so I've no idea why the latter one does not work. I'm using 3.8.14 of RabbitMQ.