alert_relabel_configs doesn't work as expected?

[Dmitry Kundo]

Hi all,

 

 I generate alerts from EC2 instances using metrics reported by node exporter. 

 For some EC2 instances, which are tagged with "low_priority_alerts: 1" tag, I'd like reduce the alert severity.

 To achieve this I'm trying to make use of the 'alert_relabel_configs', like this:

 

   alert_relabel_configs:

    - source_labels: [ low_priority_alerts ]

      regex: 1

      target_label: severity

      replacement: info

But it seems to have no effect, and the alerts are still reported with the original severity of 'error'.

Am I using it right? 

Is there another possibility to achieve what I want?

[Ian Billett]

Hi Dmitry,

Since alert labelling has the same configuration as target relabelling, you should put an example data point & relabelling rules into PromLab's amazing relabeller tool. It will show you exactly what is happening to each label and in what order.

Best,

Ian

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cbf49574-0f81-46af-a0aa-1fa003492c6fn%40googlegroups.com.

[Dmitry Kundo]

Thanks Ian,

according to the tool it should've worked. But it doesn't...

[Julius Volz]

That's odd. So it does not set the "severity" label to "info" at all on the alerts when you look at them in Alertmanager? But you do see that in the Relabeler tool? If you can reproduce it as a small test case, you could file a bug at https://github.com/prometheus/prometheus/issues/new/choose, but it would at least surprise me if that was broken (since it just uses the same relabeling as everywhere else).

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CADxb1xVuBfggg93R3OymUMruN%2BSyRwk9LPc1vffVVQOjyAT%3D3Q%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

[Julien Pivotto]

Note: The alerts are only changed when sent to alertmanager. They are
not changed in the Prometheus UI.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/cbf49574-0f81-46af-a0aa-1fa003492c6fn%40googlegroups.com.


--

Julien Pivotto
@roidelapluie