metric_relabel_config negative lookahead

[pandrian]

Hey guys im trying to basically drop a specific metric from a job that has a couple of instances but keep it only for one instance is this possible? 

i tried something like this but its not working. 

      metric_relabel_configs:
        # Drop the specific metric from all instances except primary
        - source_labels: [instance_name, __name__]
          regex: '!^primary-00,mysql_info_schema_table_size'
          action: drop

[Ben Kochie]

Negative lookahead regexp syntax is not supported.

https://github.com/google/re2/wiki/Syntax

What you can do is use use a temporary relabel to mark metrics as "keep".

Something like this:

   

metric_relabel_configs:
  # Drop the specific metric from all instances except primary
  - source_labels: [instance_name, __name__]

    regex: 'primary-00;mysql_info_schema_table_size'
    replacement: 'true'
    target_label: keep
  - source_labels: [__name__,keep]
    regex: 'mysql_info_schema_table_size;'
    action: drop
  - regex: keep
    action: labeldrop

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/46d8e23e-17b0-4f63-96cc-b8c5be3a2251n%40googlegroups.com.

[Brian Candler]

Or use "__tmp_keep" as the temporary label, then you shouldn't need the labeldrop step.

[Ben Kochie]

Oh, right, I forgot about the `__` trick.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/de5be87e-54fc-4e00-a492-34b7bedb9fb9n%40googlegroups.com.

[pandrian]

Oh wow! thanks a bunch guys that works!

Any chance you could explain step by step how does this work? i lost you on the drop action, if keep label is the set of instance and table then how does this work? im confused :D 

[Ben Kochie]

Prometheus Drop and Keep are exclusive. So they work against all metrics that match, or all metrics that don't.

What we're doing is taking a metrics like this:

mysql_info_schema_table_size{instance_name="primary-00"}
mysql_info_schema_table_size{instance_name="secondary-00"}

Phase 1 does this:

mysql_info_schema_table_size{instance_name="primary-00",__tmp_keep="true"}
mysql_info_schema_table_size{instance_name="secondary-00"}

Second phase drops where `__tmp_keep` is ""

So all you are left with is 

mysql_info_schema_table_size{instance_name="primary-00"}

(Brian's version does the labeldrop automatically)

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/2a32d9c0-f8ce-474e-978a-1fb072f303f8n%40googlegroups.com.

[pandrian]

Thanks alot Ben and Brian! really appreciated, finally can reduce the cardinality a bit :D 

[Julien Pivotto]

On 13 Jun 06:38, Brian Candler wrote:
> Or use "__tmp_keep" as the temporary label, then you shouldn't need the
> labeldrop step.

I think labeldrop is still needed. It would not be needed in regular
relabel_configs, but for metric_relabel_config it's needed.

> >> <https://groups.google.com/d/msgid/prometheus-users/46d8e23e-17b0-4f63-96cc-b8c5be3a2251n%40googlegroups.com?utm_medium=email&utm_source=footer>
> >> .

> >>
> >
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/de5be87e-54fc-4e00-a492-34b7bedb9fb9n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Brian Candler]

Thank you for that. Now you mention it, I do vaguely remember having been bitten by that before.

It might be worth a note in the documentation at

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs

where it says "It has the same configuration format and actions as target relabeling"

At

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

it then says:

Labels starting with __ will be removed from the label set after target relabeling is completed.

which is true, but on first reading I hadn't twigged that this applies only to target relabeling.