Why are so many ports attempting to be scraped?

[Scott Mattan]

I am having a problem with prometheus creating too many targets for a specific job, and I cannot understand why I cannot get the correct number of targets.

I have the following scrape configuration.

-
            job_name: kubernetes-pods
            kubernetes_sd_configs:
                - {role: pod}
            relabel_configs:
                - {action: keep, regex: true, source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]}
                - {action: replace, regex: (.+), source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path], target_label: __metrics_path__}
                - {
                     source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port],
                     action: replace,
                     regex: '([^:]+)(?::\d+)?;(\d+)',
                     replacement: '$1:$2',
                     target_label: __address__,}
                  }
                - {action: labelmap, regex: __meta_kubernetes_pod_label_(.+)}
                - {action: replace, source_labels: [__meta_kubernetes_namespace], target_label: kubernetes_namespace}
                - {action: replace, source_labels: [__meta_kubernetes_pod_name], target_label: kubernetes_pod_name}
                - {regex: '(.*):.+', replacement: '${1}', source_labels: [__address__], target_label: instance}
                - {action: replace, source_labels: [__meta_kubernetes_pod_label_cassandra_rook_io_cluster], target_label: cluster}
                - {action: replace, source_labels: [__meta_kubernetes_pod_label_cassandra_rook_io_datacenter], target_label: dc}
                - {action: replace, source_labels: [__meta_kubernetes_pod_label_cassandra_rook_io_rack], target_label: rack}

This job creates 24 targets from 3 pods with 8 ports each.  Each of the pods have the following annotations.

• prometheus.io/scrape: true
  

• prometheus.io/port: 9180
  

Due to the above config, shouldn't only one target each for a total of 3 targets be created?

maybe I am understanding the role of `prometheus.io/port` ?

If anyone has any information on this I would be grateful

[Scott Mattan]

please ignore the yaml error ->

... __address_,}

the trailing curly brace doesn't actually exist

2020年5月15日金曜日 16時50分03秒 UTC+9 Scott Mattan:

[Julius Volz]

So initially the pod discovery will give you one target for each container in a pod (see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod), and the only way to get rid of some of those targets is by dropping some of them via relabeling rules with the actions "keep" or "drop". Making two targets look the same in terms of label sets does not combine them into one target. But the only filtering you seem to do is on "__meta_kubernetes_pod_annotation_prometheus_io_scrape", which is an annotation for the whole pod (and you set it on every pod, so nothing gets filtered at all).

If you want to only keep some of the container targets within a pod, you will need to take into account one or more of the container-specific meta labels:

- __meta_kubernetes_pod_container_init: true if the container is an InitContainer
- __meta_kubernetes_pod_container_name: Name of the container the target address points to.
- __meta_kubernetes_pod_container_port_name: Name of the container port.
- __meta_kubernetes_pod_container_port_number: Number of the container port.
- __meta_kubernetes_pod_container_port_protocol: Protocol of the container port.

E.g. the container or port name could make a lot of sense to filter on.

Another (maybe better) option, if you already have Kubernetes services defined for just the endpoints you want to scrape in those pods, use the endpoint (instead of pod) discovery in Prometheus to start with a more fitting list of targets to begin with.

Btw. as an aside, none of the `prometheus.io/...` annotations have any special meaning in code. They were only used in some example configs as an inspiration for how to base scrape options on Kubernetes annotations, but are not an official or only way of doing things - the only thing that matters is that the names of the annotations fit together with the references to them in your relabeling rules.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Scott Mattan]

Julius, Thank you!

It seems that setting the annotation for `prometheus.io/port` and expecting it to magically work was an error on my part, If I wanted to use that port only I should filter to keep only that port.

That being said, I like the suggestion you gave for service endpoint scraping it really seems like it may be the better option and am going to look into it!

2020年5月15日金曜日 18時25分44秒 UTC+9 Julius Volz:

So initially the pod discovery will give you one target for each container in a pod (see https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod), and the only way to get rid of some of those targets is by dropping some of them via relabeling rules with the actions "keep" or "drop". Making two targets look the same in terms of label sets does not combine them into one target. But the only filtering you seem to do is on "__meta_kubernetes_pod_annotation_prometheus_io_scrape", which is an annotation for the whole pod (and you set it on every pod, so nothing gets filtered at all).

If you want to only keep some of the container targets within a pod, you will need to take into account one or more of the container-specific meta labels:

- __meta_kubernetes_pod_container_init: true if the container is an InitContainer
- __meta_kubernetes_pod_container_name: Name of the container the target address points to.
- __meta_kubernetes_pod_container_port_name: Name of the container port.
- __meta_kubernetes_pod_container_port_number: Number of the container port.
- __meta_kubernetes_pod_container_port_protocol: Protocol of the container port.

E.g. the container or port name could make a lot of sense to filter on.

Another (maybe better) option, if you already have Kubernetes services defined for just the endpoints you want to scrape in those pods, use the endpoint (instead of pod) discovery in Prometheus to start with a more fitting list of targets to begin with.

Btw. as an aside, none of the `prometheus.io/...` annotations have any special meaning in code. They were only used in some example configs as an inspiration for how to base scrape options on Kubernetes annotations, but are not an official or only way of doing things - the only thing that matters is that the names of the annotations fit together with the references to them in your relabeling rules.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2587bcfc-e827-4148-82d0-4d3f79277e1f%40googlegroups.com.