Relabel regex selects metrics, does not relabel

[jeff.p...@gmail.com]

I am trying to relabel metrics like this, with an app label that is coming from our cAdvisor exporter. I've removed unrelated labels for clarity:

container_cpu_system_seconds_total{container_label_marathon_app_id="active/prod/general"}

I'd like to write out a service label by taking the container_label_marathon_app_id label up until the first slash. So in the above example, service should be active,.

Here is the relabel config that I have. We first use the marathon_sd_configs to keep only the cAdvisor marathon app, and then apply the relabel:

- source_labels: ['__meta_marathon_app']

  regex:         (/cadvisor)

  action:        keep

- source_labels: [container_label_marathon_app_id]

  regex: ([^/]+).*

  target_label: service

I've read a bunch about regexes in Prometheus, along with how they are anchored, so my understanding is that this regex is turned into this under the hood:

^(?:([^/]+)).*$

That matches when I've tested the regex outside Prometheus. Additionally, if I query for metrics with the regex from my relabel, I find my expected time series:

container_cpu_system_seconds_total{container_label_marathon_app_id=~"([^/]+).*"}

Totally confused as to why the relabel action is not occurring.

I'm running Prometheus 2.0, if that helps. I can also share expanded parts of my config file if that is helpful.

[Brian Brazil]

On 26 January 2018 at 16:30, <jeff.p...@gmail.com> wrote:

I am trying to relabel metrics like this, with an app label that is coming from our cAdvisor exporter. I've removed unrelated labels for clarity:

container_cpu_system_seconds_total{container_label_marathon_app_id="active/prod/general"}

I'd like to write out a service label by taking the container_label_marathon_app_id label up until the first slash. So in the above example, service should be active,.

Here is the relabel config that I have. We first use the marathon_sd_configs to keep only the cAdvisor marathon app, and then apply the relabel:

- source_labels: ['__meta_marathon_app']

  regex:         (/cadvisor)

  action:        keep

- source_labels: [container_label_marathon_app_id]

  regex: ([^/]+).*

  target_label: service

If looks like you're mixing relabel_configs and metric_relabel_configs, see https://www.robustperception.io/relabel_configs-vs-metric_relabel_configs/

You want to put that second action in metric_relabel_configs

Brian

 

I've read a bunch about regexes in Prometheus, along with how they are anchored, so my understanding is that this regex is turned into this under the hood:

^(?:([^/]+)).*$

That matches when I've tested the regex outside Prometheus. Additionally, if I query for metrics with the regex from my relabel, I find my expected time series:

container_cpu_system_seconds_total{container_label_marathon_app_id=~"([^/]+).*"}

Totally confused as to why the relabel action is not occurring.

I'm running Prometheus 2.0, if that helps. I can also share expanded parts of my config file if that is helpful.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/120c1b1d-51c1-48b6-b063-315c1421bc0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[jeff.p...@gmail.com]

Hey Brian,

Moving my relabel config to the metric_relabel_configs fixed the problem.

That blog post was helpful, and now re-reading the relabel_config docs, I see it mentions this too -- "Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped."  I guess I hadn't internalized that distinction quite yet, heh. Anyways, thanks for the quick reply -- glad to get unstuck!