Prometheus (metric) relabel config with inverse regex match / negative lookahead

[Julian v.d Berkmortel]

Right now I'm scraping metrics from a Node Exporter. Some of the metrics which the Node Exporters exports have a `mountpoint` label.

I'd like to drop time series that have this label and **do not** match a regular expression. I tried using the `keep` action (as I'd like to keep time series that **do** match this regular expression) but this also drops all other metrics that do not have the `mountpoint` label.

"metric_relabel_configs:

  - source_labels: ['mountpoint']

    regex: '(\/home|\/var\/domains)\/something.*'

    action: keep"

I tried using the `drop` action too but this requires me to inverse the regular expression using a negative-lookahead (which isn't supported because Prometheus is written in Go of course).

What are my options in this?

**Important,** I do not have control over the way the Node Exporter is configured, thus I can't configure the Node Exporter itself to not export metrics for some specific mountpoint (if this is even possible).

[Julien Pivotto]

On 18 Jun 10:24, Julian v.d Berkmortel wrote:
> Right now I'm scraping metrics from a Node Exporter. Some of the metrics
> which the Node Exporters exports have a `mountpoint` label.
>
> I'd like to drop time series that have this label and **do not** match a
> regular expression. I tried using the `keep` action (as I'd like to keep
> time series that **do** match this regular expression) but this also drops
> all other metrics that do not have the `mountpoint` label.
>
> "metric_relabel_configs:
> - source_labels: ['mountpoint']
> regex: '(\/home|\/var\/domains)\/something.*'
> action: keep"

Can you try:

metric_relabel_configs:
- source_labels: ['mountpoint']
regex: '((\/home|\/var\/domains)\/something.*)|'

action: keep"

>
> I tried using the `drop` action too but this requires me to inverse the
> regular expression using a negative-lookahead (which isn't supported
> because Prometheus is written in Go of course).
>
> What are my options in this?
>
> **Important,** I do not have control over the way the Node Exporter is
> configured, thus I can't configure the Node Exporter itself to not export
> metrics for some specific mountpoint (if this is even possible).
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e2bf0768-8213-4c35-b0e0-4c03395ffdc6n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Aliaksandr Valialkin]

Try the following relabeling rules:

- source_labels: [mountpoint]

  regex: '(/home|/var/domains)/something.*'

  target_label: __keep

  replacement: yes

- source_labels: [mountpoint]

  regex: ''

  target_label: __keep

  replacement: yes

- source_labels: [__keep]

  regex: yes

  action: keep

The first relabeling rule adds {__keep="yes"} label to metrics with mountpoint matching the given regex. The second relabeling rule adds {__keep="yes"} label to metrics with empty `mountpoint` label, e.g. metrics without this label. The last relabeling rule drops all the metrics without {__keep="yes"} label. See https://www.robustperception.io/or-in-relabelling .

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e2bf0768-8213-4c35-b0e0-4c03395ffdc6n%40googlegroups.com.

--

Best Regards,

Aliaksandr Valialkin, CTO VictoriaMetrics

[Julian v.d Berkmortel]

Thank you, this works like a charm!

Op vrijdag 18 juni 2021 om 19:57:42 UTC+2 schreef Julien Pivotto:

[Julien Pivotto]

On 21 Jun 12:20, Julian v.d Berkmortel wrote:
> Thank you, this works like a charm!
>
> Op vrijdag 18 juni 2021 om 19:57:42 UTC+2 schreef Julien Pivotto:
>
> > On 18 Jun 10:24, Julian v.d Berkmortel wrote:
> > > Right now I'm scraping metrics from a Node Exporter. Some of the metrics
> > > which the Node Exporters exports have a `mountpoint` label.
> > >
> > > I'd like to drop time series that have this label and **do not** match a
> > > regular expression. I tried using the `keep` action (as I'd like to keep
> > > time series that **do** match this regular expression) but this also
> > drops
> > > all other metrics that do not have the `mountpoint` label.
> > >
> > > "metric_relabel_configs:
> > > - source_labels: ['mountpoint']
> > > regex: '(\/home|\/var\/domains)\/something.*'
> > > action: keep"
> >
> > Can you try:
> >
> > metric_relabel_configs:
> > - source_labels: ['mountpoint']
> > regex: '((\/home|\/var\/domains)\/something.*)|'
> > action: keep"

Since I forgot to explain it:

An empty label and a missing label are the same in prometheus.

By isolating the first part of the regex with () and adding | at the end
of the regex, we make sure that it matches the first part OR it is
missing (regexes are anchored in prometheus). The added | at the end
means "or empty").

> >
> > >
> > > I tried using the `drop` action too but this requires me to inverse the
> > > regular expression using a negative-lookahead (which isn't supported
> > > because Prometheus is written in Go of course).
> > >
> > > What are my options in this?
> > >
> > > **Important,** I do not have control over the way the Node Exporter is
> > > configured, thus I can't configure the Node Exporter itself to not
> > export
> > > metrics for some specific mountpoint (if this is even possible).
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "Prometheus Users" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > an email to prometheus-use...@googlegroups.com.
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-users/e2bf0768-8213-4c35-b0e0-4c03395ffdc6n%40googlegroups.com
> > .
> >
> >
> > --
> > Julien Pivotto
> > @roidelapluie
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/13f6d9be-0db2-4bee-8098-5ffc04e4dd04n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie