Prometheus & Kubernetes - Scrape Annotations

[ryan woods]

Hey guys,

Currently I have Prometheus deployed on kube with auto discovery enabled and working as expected, but I'm wondering if it's at all possible for Prometheus to scrap metadata/labels from Kube objects? For example

apiVersion: apps/v1

kind: Deployment

metadata:

  name: alertmanager

  namespace: monitoring

spec:

  replicas: 1

  selector:

    matchLabels:

      app: alertmanager

  template:

    metadata:

      name: alertmanager

      labels:

        app: alertmanager

        customer: test_customer

Is Prometheus able to scrap the customer label and 'tag' the metrics collected from this Deployment with it?

Thanks,

Ryan

[Julius Volz]

Hi Ryan,

Yes, you can use the "labelmap" relabeling action to map Kubernetes object labels into target labels (and thus later on, time series labels). Here is one example of that for endpoints scraped on Kubernetes:

https://github.com/prometheus/prometheus/blob/cd12f0873c3eb2031f7ba9b2e169449aa1012e3f/documentation/examples/prometheus-kubernetes.yml#L168-L169

This relabeling rule takes all hidden meta labels (only visible between service discovery and the relabeling phase) that start with "__meta_kubernetes_service_label_XXX" and maps the XXX part into permanent target labels.

How do we know we need __meta_kubernetes_service_label_XXX? That's documented for the k8s endpoints SD here: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#endpoints

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7da8c934-cb0b-4591-b6f3-92e26daed76e%40googlegroups.com.