How do I to determine if a single alert out of a set of grouped together alerts common label has a certain value?

[a z]

Hi,

 

I am in the process of trying to have alerts generated by AlertManager to be service focused.

 

What I am trying to do is group alerts by service (for example group database server, web server, HTTP poles together if they belong to the same service/application and send a single alert contain all triggered rules. I have this part working without any issues.

 

What I now want to do is improve the alert template and control the banner colour based on another label which will be common among all alerts. For example, each triggered rule/alert also has a severity which can be “critical”, “waring”, “info”.  If any of the alerts are “critical” I want to set the banner to red, if there are no critical alerts, but there is a warning then I want to set the banner to orange. Finally if there are no warning or critical alerts we set the banner to yellow.

 

The intention by this is so we can scan though slack / emails and just see well this service has 5 triggered alerts but its yellow so nothing critical, however this other service is red with 1 alert, meaning we need to jump on the second service due to it having a more critical issue first.

 

I am unsure how within Prometheus/Alertmanager templating how I can check if one of the alert labels has a certain value.

 

Does anyone have any suggestions?

[Christian Hoffmann]

On 7/16/20 11:51 PM, 'a z' via Prometheus Users wrote:
> I am unsure how within Prometheus/Alertmanager templating how I can
> check if one of the alert labels has a certain value.

In Alertmanager templates, you can use Go's template syntax which allows
for the "eq" function. You can see an example here:

https://github.com/prometheus/alertmanager/blob/master/template/default.tmpl#L4

If you are grouping by this label, you can access it using .GroupLabels:

https://prometheus.io/docs/alerting/latest/notifications/

Kind regards,
Christian

[Lakshman Savadamuthu]

Thanks a lot for the assistance Christian, after checking various options, finally decided to reboot the host.

After reboot, the service was back online.

Thanks again for your help,

Lakshman

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/250e788c-11ad-d1e8-cf06-2d0aea65d226%40hoffmann-christian.info.