Prometheus agent, hashmod sharding issue

[Vikas Budhvant]

We have 4 prom-agents having scrape config as below
regrex is different for each agent(0, 1, 2, and 3)

` - job_name: 'kube-pods'

kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__address__] modulus: 4 target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ^0$ action: keep - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name`

our expectations was same metrics should not be scraped by 2 prometheus agent, which is not working and we are seeing same metrics being scraped by more than 1 agents.
I verified this using query
count(count({job="kube-pods"}) by (prometheus_agent,kubernetes_pod_name,name,instance)) by (kubernetes_pod_name,name,instance) > 1

there are some metrics with exactly 1 count as well which means issue is not consistent for all metrics. not sure if its something to do with pods or their config or somethings else.

Interestingly we don't see such duplicated for below scrape job
` - job_name: 'hlo-pods'

kubernetes_sd_configs: - role: pod namespaces: names: - hlo - relabel_configs: - source_labels: [__address__] modulus: 4 target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ^2$ action: keep - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - source_labels: [ __meta_kubernetes_pod_container_name ] action: replace target_label: kubernetes_container_name `

Any clue what should I check? what could be possible cause?

[Julien Pivotto]

This indeed should not happen. Maybe can throw more than the address
label into the hash and also include the port?

On 23 Jan 02:36, Vikas Budhvant wrote:
>
>
> We have 4 prom-agents having scrape config as below
> regrex is different for each agent(0, 1, 2, and 3)
>
> ` - job_name: 'kube-pods'
> kubernetes_sd_configs: - role: pod relabel_configs: - source_labels:
> [__address__] modulus: 4 target_label: __tmp_hash action: hashmod -
> source_labels: [__tmp_hash] regex: ^0$ action: keep - source_labels:
> [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex:
> true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
> action: replace target_label: __metrics_path__ regex: (.+) - source_labels:
> [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action:
> replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label:
> __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) -
> source_labels: [__meta_kubernetes_namespace] action: replace target_label:
> kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action:
> replace target_label: kubernetes_pod_name`
>
> our expectations was same metrics should not be scraped by 2 prometheus
> agent, which is not working and we are seeing same metrics being scraped by
> more than 1 agents.
> I verified this using query
> count(count({job="kube-pods"}) by (prometheus_agent,kubernetes_pod_name,

> *name*,instance)) by (kubernetes_pod_name,*name*,instance) > 1

>
> there are some metrics with exactly 1 count as well which means issue is
> not consistent for all metrics. not sure if its something to do with pods
> or their config or somethings else.
>
> Interestingly we don't see such duplicated for below scrape job
> ` - job_name: 'hlo-pods'
> kubernetes_sd_configs: - role: pod namespaces: names: - hlo -
> relabel_configs: - source_labels: [__address__] modulus: 4 target_label:
> __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ^2$ action:
> keep - action: labelmap regex: __meta_kubernetes_pod_label_(.+) -
> source_labels: [__meta_kubernetes_namespace] action: replace target_label:
> kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action:
> replace target_label: kubernetes_pod_name - source_labels: [
> __meta_kubernetes_pod_container_name ] action: replace target_label:
> kubernetes_container_name `
>
> Any clue what should I check? what could be possible cause?
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4b460268-5bd0-41af-88ef-d42ba7315785n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie