Uptime when no up metric is available

[Tom Gregory]

I'm using Prometheus federation i.e. I'm ingesting metrics from one Prometheus to another.

To do some relabelling I'm using metric_relabel_configs. According to the docs:

Metric relabelling does not apply to automatically generated timeseries such as up.  

That makes me sad because I want to measure uptime using these metrics. Is there any suggested approach to getting around this problem please? Any way to measure uptime without the automatically generated up metric?

[Stuart Clark]

On 2021-01-13 13:55, Tom Gregory wrote:
> I'm using Prometheus federation i.e. I'm ingesting metrics from one
> Prometheus to another.
>

> To do some relabelling I'm using _METRIC_RELABEL_CONFIGS_. According
> to the docs [1]:
>
> METRIC RELABELLING DOES NOT APPLY TO AUTOMATICALLY GENERATED
> TIMESERIES SUCH AS UP.

>
> That makes me sad because I want to measure uptime using these
> metrics. Is there any suggested approach to getting around this
> problem please? Any way to measure uptime without the automatically

> generated _up_ metric?
>

What is the problem you are having with the up metric?

up (and a few others) are created by Prometheus to give some details
about scrapes. So they should "just work".

--
Stuart Clark

[Tom Gregory]

The relabelling process doesn't happen on the up metric, so the metric doesn't have the correct labels I need for my query.

To illustrate this, imagine I have two services generating metrics

• service A generates metrics with a label svc="service-a"
• service B generates metrics with a label svc="service-b"

For whatever reason I want to relabel svc to service. This works for all the metrics other than up. 

I want to run Prometheus queries against different metrics using a consistent set of labels (e.g. service rather than svc). This is especially important using visualisation tools like Grafana (imagine a service drop down box).

[Ben Kochie]

That sounds like metric_relabel_configs, not relabel_configs. relabel_configs apply to `up` and other auto-generated metrics.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2b55e54f-d0cb-493d-ae69-0ef02bcf4edcn%40googlegroups.com.

[Stuart Clark]

On 2021-01-13 16:11, Tom Gregory wrote:
> The relabelling process doesn't happen on the _up _metric, so the

> metric doesn't have the correct labels I need for my query.
>
> To illustrate this, imagine I have two services generating metrics
>

> * service A generates metrics with a label svc="service-a"
> * service B generates metrics with a label svc="service-b"
>
> For whatever reason I want to relabel _svc_ to _service_. This works
> for all the metrics other than _up_.

>
> I want to run Prometheus queries against different metrics using a

> consistent set of labels (e.g. _service _rather than _svc_). This is

> especially important using visualisation tools like Grafana (imagine a

> _service _drop down box).
>

Are the metrics from those two services coming from the same scrape or
separate scrapes?

The up metric gives details per-scrape, so the labels attached to it are
based on the service discovery relabling process. Is that how you are
adding the "svc" label?

--
Stuart Clark

[Tom Gregory]

Hi Stuart. The metrics are coming from different scrapes. Each service is a different endpoint.

I'm doing the relabelling using metric_relabel_configs.

Is this the right approach?

Regards,

Tom Gregory

[Stuart Clark]

On 13/01/2021 20:31, Tom Gregory wrote:

Hi Stuart. The metrics are coming from different scrapes. Each service is a different endpoint.

I'm doing the relabelling using metric_relabel_configs.

metric relabeling is really more for removing metrics (if you can't turn them off on the exporter/application side).

For adding target specific tags you want to use "relabel_config" which is run as part of the service discovery process. That's where you might add a service label, and it would apply to all the metrics scraped as well as the automatic ones such as "up".

[Tom Gregory]

Thanks Stuart. I switched to relabel_config and now all good.