Alertmanager sent multiple empty mails when template rendering failed

[Theofanis Pispirigkos]

Hi , in my case a rendering error caused alertmanager to send multiple empty emails. I am using prometheus operator with 2 alertmanagers in version v0.18.1

The alertmanager configuration looks like:

config:
  route:
    routes:
    - match:
        severity: ticket
      group_by: ['...']
      group_wait: 1s
      group_interval: 1s
      repeat_interval: 10d
      receiver: ticket
  receivers:
    - name: 'ticket'
      email_configs:
        - to: '....'
          headers:
            subject: 'Prometheus alerts'
          html: >-
            {{ range .Alerts }}
              <pre>
              title: {{ .Annotations.summary }} ({{ .Labels.cluster }})
              assignee: {{ .Annotations.assignee }}
             {{ end }}
      send_resolved: true

The error occured when i used .GroupLabels.cluster instead of .Labels.cluster . Alertmanager logged the bellow messages and sent multiple empty mails.

level=error ts=2020-02-18T11:17:42.017Z caller=notify.go:367 component=dispatcher msg="Error on notify" err="cancelling notify retry
 for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.GroupLabels.k8scluster>: can
't evaluate field GroupLabels in type template.Alert" context_err=null

level=error ts=2020-02-18T11:17:42.017Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err=
"cancelling notify retry for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.Grou
pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert"

level=error ts=2020-02-18T11:17:43.020Z caller=notify.go:367 component=dispatcher msg="Error on notify" err="cancelling notify retry
 for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.GroupLabels.k8scluster>: can
't evaluate field GroupLabels in type template.Alert" context_err=null

level=error ts=2020-02-18T11:17:43.020Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err=
"cancelling notify retry for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.Grou
pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert"

level=error ts=2020-02-18T11:17:44.020Z caller=notify.go:367 component=dispatcher msg="Error on notify" err="cancelling notify retry
 for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.GroupLabels.k8scluster>: can
't evaluate field GroupLabels in type template.Alert" context_err=null

level=error ts=2020-02-18T11:17:44.020Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err=
"cancelling notify retry for \"email\" due to unrecoverable error: execute html template: template: :13:32: executing \"\" at <.Grou
pLabels.k8scluster>: can't evaluate field GroupLabels in type template.Alert"

Since the emails are routed to a ticketing system , having the alertmanager spamming with empty emails is not an acceptable behavior. So my questions is why alertmanager sent eventually the email and why it kept firing emails until the time i fixed the configuration.  Is there any way to prevent this situation from happening ?

[Simon Pasquier]

See https://github.com/prometheus/alertmanager/issues/1944

A group_interval parameter of 1sec isn't really recommended and
explains why Alertmanager sends emails so frequently.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/eca83f70-7ab0-42c3-8ce1-887285146323%40googlegroups.com.