missing custom labels of Prometheus in PagerDuty alerts

[NEEL THOMAS]

I have a Prom CPU alert rule defined like below :

- alert: HostHighCpuLoad expr: (sum by (instance) (avg by (mode, instance) (rate(node_cpu_seconds_total{mode!="idle"}[2m]))) > 0.8) * on(instance) group_left (nodename) node_uname_info{nodename=~".+"} for: 0m labels: severity: warning product : backend-app annotations: summary: Host high CPU load (instance {{ $labels.instance }}) description: "CPU load of {{ $labels.product }} is greater than > 80%"

its triggering  the alert, but the value of custom label product : backend-app  is not interpolating in the alert description part, its coming like below

"CPU load of is greater than > 80%"

expected is

"CPU load of backend-app is greater than > 80%"

any idea why the custom label is not picking up, although the alert is getting triggered

or how can we add custom alert labels that are accessible in annotation.

thanks

[Brian Candler]

I'm not sure, but I don't think that labels added *within* the alerting rule are accessible to annotations within the same alerting rule. For for "{{ $labels.product }}", you can simply replace it with the literal value, i.e. "backend-app"

[Charls P John]

Hi Neel :)

I think the custom labels added in the alert rule is meant for referred by downstream apps like alertmanager. One way is to add this label in the scraping configuration for node_exporter itself (relabel_config:) and mention this label inside avg by (mode, instance, product)

---

Best Regards,

Charls P John

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/86ca48d4-71be-46f9-9a1e-80342d94c212n%40googlegroups.com.