double amount of metrics when using federation endpoint with two replicas

[Veera]

I try to scrape from a federation endpoint, which resides inside an open shift cluster. The openshift-Prometheus-instance has two replicas. I am getting the double amount of metrics, labeled with "prometheus_replica=replica_0" and "prometheus_replica=replica_1".

I would be great, if I could filter the replicas in my scrape-config.

• job_name: prometheus_openshift_prod
  honor_labels: true
  metrics_path: '/federate'
  params:
  'match[]':
  - '{job="kube-state-metrics"}'
  - '{job="kubelet"}'
  - '{job="kube-controllers"}'
  - '{job="apiserver"}'
  - '{job="node-exporter"}'
  - '{job="kubernetes-apiservers"}'
  - '{name=~"job:.*"}'

I use Version 2.15.0
The problem is also described here:

openshift/telemeter#151

[Stuart Clark]

On 2020-04-14 15:49, Veera wrote:
> I try to scrape from a federation endpoint, which resides inside an
> open shift cluster. The openshift-Prometheus-instance has two
> replicas. I am getting the double amount of metrics, labeled with
> "prometheus_replica=replica_0" and "prometheus_replica=replica_1".
>
> I would be great, if I could filter the replicas in my scrape-config.
>

Rather than scraping each Prometheus server directly you want to use a
loadbalancer/service so it only connects to one of the two

> * job_name: prometheus_openshift_prod

> honor_labels: true
> metrics_path: '/federate'
> params:
> 'match[]':
> - '{job="kube-state-metrics"}'
> - '{job="kubelet"}'
> - '{job="kube-controllers"}'
> - '{job="apiserver"}'
> - '{job="node-exporter"}'
> - '{job="kubernetes-apiservers"}'
> - '{name=~"job:.*"}'
>
> I use Version 2.15.0
> The problem is also described here:
>

> openshift/telemeter#151 [1]
>
> --
> You received this message because you are subscribed to the Google
> Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/prometheus-users/a5456e01-07ab-401c-9702-621474bcc837%40googlegroups.com
> [2].
>
>
> Links:
> ------
> [1] https://github.com/openshift/telemeter/issues/151
> [2]
> https://groups.google.com/d/msgid/prometheus-users/a5456e01-07ab-401c-9702-621474bcc837%40googlegroups.com?utm_medium=email&utm_source=footer

--
Stuart Clark

[Veera]

• job_name: prometheus_openshift_prod
  honor_labels: true
  metrics_path: '/federate'
  params:
  'match
  

• job_name: prometheus_openshift_prod
  honor_labels: true
  metrics_path: '/federate'
  params:
  'match[]':
  - '{job="kube-state-metrics"}'
  - '{job="kubelet"}'
  

• scheme: https

        static_configs:
         -targets:
           - endpoint

we made a config file in these way also  we are getting double amount of metrics

[Veera]

• job_name: prometheus_openshift_prod
• honor_labels: true
• metrics_path: '/federate'
• params:
• 'match[]':

• '{job="kube-state-metrics"}'

• '{job="kubelet"}'
• scheme: https
• static_configs:
• -targets:
• - endpoint

we made a change add endpoint to target even though it was getting double end points 
Thanks,
Veera

> an email to promethe...@googlegroups.com.