How to get UP metrics of prometheus which is running in federate mode

[Venkata Sandeep]

Hi All,

We have multiple prometheus instances, each scraping the metrics of different services

We have a federated prometheus to scrape metrics from these multiple prometheus instances (with honor_labels as true). 

If one of the prometheus instance is down, federated prometheus does n't have any metrics of the actual service. So it is becoming hard to understand if the prometheus instance is down or if the actual service is down.

How to get the UP metrics of prometheus instance into federated prometheus?

Thanks

Sandeep

[Julius Volz]

The federation scrapes themselves also generate "up" metrics. So e.g. if your federation scrape config is:

  - job_name: federate
    honor_labels: true
    metrics_path: '/federate'
    static_configs:
      - targets:
        - other-prometheus:9090

Then you will get up{job="federate", instance="other-prometheus:9090"} set to 0 or 1 based on whether the federation scrape failed or succeeded. honor_labels doesn't prevent this, the only time it makes things behave differently is when target labels of a scrape target are already present in the scraped metrics, in which case honor_labels keeps the labels sent by the target instead of overwriting them with the target labels.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/459b4817-6692-43c6-a1ec-efe0b6f3950a%40googlegroups.com.