how to reverse a match in metric_relabel_config

[Khusro Jaleel]

Hi there, I've got a job that does service discovery for "pods" in kubernetes and generates metrics like the following:

http_server_request_duration_seconds_bucket{path="/path1",pod="podName"}

http_server_request_duration_seconds_bucket{path="/path2",pod="podName"}

http_server_request_duration_seconds_bucket{path="/path3",pod="podName"}

http_server_request_duration_seconds_bucket{path="foo",pod="podName"}

http_server_request_duration_seconds_bucket{path="bar23324",pod="podName"}

http_server_request_duration_seconds_bucket{path="baz",pod="podName"}

http_server_request_duration_seconds_bucket{path="something completely random here",pod="podName"}

http_server_requests_total{path="/path5",pod="someOtherPod"}

http_server_requests_total{path="/path6",pod="someOtherPod"}

inf_metric{name="foo",pod="someOtherPod"}

some_other_metric_count{name="bar",pod="someOtherPod2"}

I want to DROP any metrics that match a certain pod and the metric name "http_server_request_duration_seconds_bucket " and DON'T contain the following values for the "path" label: "path1, path2, path3". These are the only values I want to "keep" so to speak.

I'm not sure how to accomplish this using a "metric_relabel_config", because if I match on "^http_server_request_duration_seconds_bucket;podName;.*path1|.*path2|.*path3$' and tell it to "keep" the values, it throws away ALL the other metrics for ALL other pods and keeps only a very small set. This is obviously not what I want. I want to keep the metrics for all of my pods, but I want to drop a small set of metrics from a certain pod that DON'T match the regex I've specified and I'm not sure how to do this. 

I'm trying something like this as an example, which doesn't work - it deletes ALL other metrics for all other pods.

  - job_name: 'k8s-pods'

    scrape_interval: 30s

    metric_relabel_configs:

    - source_labels: [ __name__, pod, path ]

        regex: ^http_server_request_duration_seconds_bucket;podName;.*path1|.*path2$

        action: keep

Ideally I'd like to use a "drop" apply a "!" or a "NOT" type operator to the regex if that's possible?

[Brian Brazil]

Generally metric_relabel_configs should only be used as a stopgap, until you can fix the job - which is definitely the best option here.

To do this in RE2 you need something like .*([^p]|p[^a]|pa^[t]|pat[^h]|path[^12])$ with a drop action.

--

Brian Brazil

www.robustperception.io

[Khusro Jaleel]

Thanks Brian! Unfortunately in this case we are using a third party rails library and can't easily see how to tell it to stop generating certain values for the "path=" label and only generate metrics for the valid ones. So I'm having to use a really ugly workaround. Would it be possible to add this as an option into Prometheus itself though in the future, i.e. a NOT match regex type option? 

[Brian Brazil]

On Fri, 13 Sep 2019 at 08:58, Khusro Jaleel <kerne...@gmail.com> wrote:

Thanks Brian! Unfortunately in this case we are using a third party rails library and can't easily see how to tell it to stop generating certain values for the "path=" label and only generate metrics for the valid ones. So I'm having to use a really ugly workaround. Would it be possible to add this as an option into Prometheus itself though in the future, i.e. a NOT match regex type option? 

I can't see us adding this to be honest, relabelling is already pretty complicated for users and cases it doesn't cover are rare.

--

Brian Brazil

www.robustperception.io

[Khusro Jaleel]

OK no worries. I'm thinking that another way to get around this is to allow the initial job to completely drop ALL metrics for "http_server_request_duration_seconds_bucket" and then create another job that only "keeps" a small amount of metrics, i.e. "path1, path2, path3".

[Christian Hoffmann]

Hi,

On 2019-09-13 01:09, Khusro Jaleel wrote:
> I want to DROP any metrics that match a certain pod and the metric name
> "http_server_request_duration_seconds_bucket " and DON'T contain the
> following values for the "path" label: "path1, path2, path3". These are
> the only values I want to "keep" so to speak.

I think it may work if you split this into two steps. First, mark those
metrics you want to keep with a temporary label, then drop all metrics
with that name which don't have it.

E.g.:

metric_relabel_configs:
- source_labels: [__name__, pod, path]
regex:
"^http_server_request_duration_seconds_bucket;podName;.*path1|.*path2$"

action: "replace"
target_label: "__tmp_keep"
replacement: "keepme"

- source_labels: [__name__, __tmp_keep]
regex: "^http_server_request_duration_seconds_bucket;$"
action: "drop"

Can you try this? (haven't tested)

Kind regards
Christian

[Khusro Jaleel]

Thanks for that tip Christian, I was thinking of doing the same but had not tried it yet. I did try it and it works perfectly, so thank you :-)