Kubernetes & Service Discovery: endpoints vs service role

[Patrick Schless]

I'm using prometheus on k8s, and trying to add an annotation to my service for the service discovery. The metrics I'm exposing are business-level (eg "# of customers") so I don't need all pods queried. Querying the service once (with a random pod) is sufficient. As I understand it, there is a service discovery role applicable: "service".

However, when I add this annotation to my service, it is "discovered" by the endpoints role, which then tries to hit all downstream pods.

prometheus.io/scrape: true

The docs clearly make a distinction between service SD and endpoints SD, but it's not clear to me now to choose one or the other.

[Matthias Rampke]

Kubernetes creates the Endpoints object based on the Service object. Does it carry over the annotations while doing that? If so, you are getting returns in both SD configs because there are two objects in Kubernetes to be discovered.

I would suggest using a different value for the scrape annotation, and matching on that in the service SD configuration, to distinguish whether you want this thing scraped once or for each instance.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/bbec2f68-afea-4366-8156-b729bc04b373%40googlegroups.com.

[Patrick Schless]

Hi Matthias,

Thanks for your thoughts.  I've experimented some more, and have it working now.  Best as I can tell, it's simply missing from the default config in the prom/prometheus docker image: https://gist.github.com/plainlystated/95c79163583d5d107f2d6e85f9d335d5

In that config, there are only two "role: service" sections.  One is the pushgateway, and the other forces use of the blackbox app, which seems like it shouldn't be necessary for what I'm trying to do.

I added a scrape config, which seems to have done the trick:

    - honor_labels: true
      job_name: prometheus-service-non-blackbox
      kubernetes_sd_configs:
      - role: service
      relabel_configs:
      - action: keep
        regex: service
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__

Then I just set the service annotation:

prometheus.io/scrape: service

Now my prometheus correctly has 1 target for the service, and no targets for the endpoints.

Thanks for your help!

- Patrick

[Matthias Rampke]

Question to the wider audience: would it be sensible to add this to the example/default Kubernetes configuration?

/MR

[Patrick Schless]

My two cents:

I have a lot of metrics to monitor which aren't specific to the individual pods.  Having the service-role supported (non-"blackbox") in the default configs would've saved me a lot of confusion, and I bet I'm not alone.  Failing that, calling this use case out more clearly in the docs would've bee helpful.  I'm new to prometheus and unfamiliar with its configs, so the solution wasn't as trivial for me as it would be for some.