change mail template on windows

[Kolja Krückmann]

Hi y'all

I'm trying to change the way the mail from my alertmanger looks and behaves.

My goal is to have the subject list the alertname and the targets detected by that alert. (I need it like that for future todo's)

My first problem is, i dont have any templates in my files. Do I need to clone one from git? How does the prom know, I have a template? 

The second one would be to change the subject in order for my requirements to fit.

Kind regards,

Kolja

[Kolja Krückmann]

Currently my alertmanager.yml looks like this:

receivers:
- name: 'email'
  email_configs:
  - to: 'X...@company.com'
    send_resolved: true
    headers:
      Subject: '{{ .CommonLabels.alertname }} - {{ .CommonLabels.instance }}'

Unfortunately the subject in the mail now only displays both labels if only there is an alert for a single instance (like High CPU Usage on core 0,1,2,3 on one instance (see screenshot attatched)) As soon as the alert is for more then just one instance the instance lable is completly missing... Why is it missing? And how can I change it to display all instances affected by that alert.

Kind regards.
Kolja

[Brian Candler]

As its name implies, "CommonLabels" contains only those labels which are common to all alerts in the group.  If there are multiple instances, then the "instance" label is not common to all alerts, so won't be in this object.

You want to iterate over "Alerts" (or "Alerts.Firing") and look at the Labels.instance within each alert. You will find an example showing how to do that here:

https://prometheus.io/docs/alerting/latest/notification_examples/#ranging-over-all-received-alerts

Additional references:

https://prometheus.io/docs/alerting/latest/notifications/#data

https://prometheus.io/docs/prometheus/latest/configuration/template_reference/

[Kolja Krückmann]

Hi all, I tried it now for two days straight and I just don't get it working.

First of all where do I actually find the expressions I can use to define the subject under the header? I am still missing a template I can use? I don't really have one in my Prom/Alert-files.
Second of all my current receivers are configured like so:

receivers:
- name: 'email'
  email_configs:

  - to: 'X...@XXX.com'
    send_resolved: true
    headers:
      Subject: "{{ .Alerts.alertname }} - {{ range .Alerts }} {{ .Alerts.instance }} {{ end }}"

Like Brian said, I want to iterate in the second half of that "statement" (I don't actually know how these {{ .xyz }} are called) through all instances and the first one should just say which alert is now firing.

I also tried it as followed:

Subject: "{{ .GroupLabels.alertname }} - {{ range .Alerts }} {{ .Alerts.instance }} {{ end }}"

and

Subject: "{{ .Labels.alertname }} - {{ range .Alerts }} {{ .Alerts.instance }} {{ end }}"

but nothing is really working.

All I really want is the Subject like so e.x.:

InstanceDown - SVR-DS01 SVR-DC11
       ^^^^^^^^            ^^^^^^^^    ^^^^^^^^          

    Alertname    -   Instance1 Instance2 (if it is also possible to seperate the instances via comma would be perfect)

[Kolja Krückmann]

[EDIT]
Also I want to split the firing and resolved alerts.

[Kolja Krückmann]

Got it working like this:

receivers:
- name: 'email'
  email_configs:

  - to: 'sysw...@XXX.com'
    send_resolved: false
    headers:
      subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}'

- name: 'email_warning'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: false
    headers:
      subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}'

- name: 'email_resolved'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: true
    headers:
      subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts }}{{ .Labels.instance }}, {{ end }}'

[Brian Candler]

Maybe you want .Alerts.Firing and .Alerts.Resolved.  A single alert group is likely to contain a mixture of firing alerts and resolved alerts.

Also, note that your "send_resolved: true" receiver doesn't only send resolved messages: it will send on firing *and* on resolved. Hence the subject "Resolved: ..." is likely to be very misleading.

[Kolja Krückmann]

I changed the config to this:

- name: 'email_resolved'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: true
    headers:

      subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'

But this is sending absolutly nothing. What did I do wrong?

[Kolja Krückmann]

Or do I need to change it to explicitly say in the crit and warn mail to range over the .Alerts.Firing in order to get the third receiver to work?

receivers:
- name: 'email'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: false
    headers:

      subject: 'Critical: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'

- name: 'email_warning'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: false
    headers:

      subject: 'Warning: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts.Firing }}{{ .Labels.instance }}, {{ end }}'

- name: 'email_resolved'
  email_configs:
  - to: 'sysw...@XXX.com'
    send_resolved: true
    headers:

      subject: 'Resolved: {{ .GroupLabels.alertname }} - Instances: {{ range .Alerts.Resolved }}{{ .Labels.instance }}, {{ end }}'

[Brian Candler]

You'll need to show the rest of your alertmanager config. "Receivers" by themselves don't do anything; you have to refer to them in routing rules.

[Kolja Krückmann]

Hi Brian, thanks here are the routes to the receivers:

  routes:
  - receiver: email_critical
    group_interval: 1m
    group_wait: 5m
    repeat_interval: 24h
    group_by: ['alertname', 'severity']
    matchers:
    - severity="critical"
    - severity!="resolved"
 
  - receiver: email_warning
    group_interval: 10m
    group_wait: 5m
    repeat_interval: 24h
    group_by: ['alertname', 'severity']
    matchers:
    - severity="warning"
    - severity!="resolved"

  - receiver: email_resolved
    match_re:
      severity: ^(resolved)$

the match_re part from resolved is from good ol' friend chatgpt

[Brian Candler]

That will never work.  All labels (including severity) are things set in your alerting rules.  You will never see a label with severity=resolved, unless you actually make an alerting rule which sets severity=resolved when it fires.

(match_re does work though; it's the old syntax whereas "matchers" is the new and preferred syntax)

Moral: never trust what ChatGPT tells you.

Off the top of my head, I can't think of any way to send a message when alerts are resolved but *not* when alerts are firing.

There are good reasons for not sending *any* resolved messages though:

https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped

[Kolja Krückmann]

Thank you very much Brian!

This will be my last post for now. I'm going to change company. I hope you can help my collegues as good as you helped me!

This community is great, keep it up.