Unexpected EOF for AlertManager Template

[Romenyrr]

Greetings, 

I'm trying to configure a notification template for Opsgenie, but for somereason Alertmanager refuses to take it. It keeps saying that there's an unexpected EOF, but I believe I've followed the correct Go templating rules. I've tried wrapping the expression in quotes but no luck. If I use this rule in the alertmanager.yml file, it works just fine. I'm stumpped =/... Has anyone seen this before? 

Here's the rule in my 'myorg.tmpl' file:

{{ define "opsgenie.myorg.priority" }} {{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else}}P3{{ end }} {{ end }}

Here's the error: 

level=info ts=2020-01-02T22:08:08.072Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=alertmanager.yml

ts=2020-01-02T22:08:08.084Z caller=coordinator.go:137 component=configuration msg="one or more config change subscribers failed to apply new config" file=alertmanager.yml err="failed to parse templates: template: myorg.tmpl:1: unexpected EOF"

AlertManager Info

Version Information

Branch:

HEAD

BuildDate:

20191211-14:13:14

BuildUser:

root@00c3106655f8

GoVersion:

go1.13.5

Revision:

f74be0400a6243d10bb53812d6fa408ad71ff32d

Version:

0.20.0

[Romenyrr]

I'm getting closer, as I'm not getting the error now (I was missing an {{ end }} ) when alertmanager starts up, but when I tries to process the routing rule, I get a new error: 

level=error ts=2020-01-03T05:36:34.129Z caller=dispatch.go:301 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"opsgenie\" due to unrecoverable error: unexpected status code 422: {\"message\":\"Request body is not processable. Please check the errors.\",\"errors\":{\"priority\":\"should be one of [ P1, P2, P3, P4, P5 ]\"},\"took\":0.001,\"requestId\":\"c4589a15-34e6-4530-8dde-d45f3557f6f4\"}"

Cleaned up Define block: 

{{ define "opsgenie.myorg.priority"}} 

{{ range .Alerts }}

{{ if eq .Labels.severity "critical"}}

P1

{{ else }}

P3

{{ end }} 

{{ end }}

{{ end }}

[Brian Candler]

In alertmanager's opsgenie_config, what have you put for the "priority: ..." line?

[Simon Pasquier]

If you have many alerts, then the output string would be "P1P1..."
which isn't valid for OpsGenie.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/94b64886-3f90-4ab6-9baa-ba69a5f6d34a%40googlegroups.com.

[Romenyrr]

@brian, Sorry that I forgot to include the Opsgenie config, but here it is below: 

@simon, I'm just testing this for a single alert at the moment, but I'll actually take a look. 

  - name: "opsgenie"

    opsgenie_configs:

      - api_key: 12345

        message: "{{ range .Alerts }} {{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"

        send_resolved: true

        description: '{{ template "opsgenie.myorg.description" . }}'

        note: '{{ template "opsgenie.myorg.note" . }}'

        tags: '{{ template "opsgenie.myorg.tags" . }}'

        priority: '{{ template "opsgenie.myorg.priority" . }}'

On Thursday, January 2, 2020 at 2:11:19 PM UTC-8, Romenyrr wrote:

[Christian Hoffmann]

Hi,

On 1/3/20 6:41 AM, Romenyrr wrote:
> |
> level=error
> ts=2020-01-03T05:36:34.129Zcaller=dispatch.go:301component=dispatcher
> msg="Notify for alerts failed"num_alerts=1err="cancelling notify retry

> for \"opsgenie\" due to unrecoverable error: unexpected status code 422:
> {\"message\":\"Request body is not processable. Please check the
> errors.\",\"errors\":{\"priority\":\"should be one of [ P1, P2, P3, P4,
> P5
> ]\"},\"took\":0.001,\"requestId\":\"c4589a15-34e6-4530-8dde-d45f3557f6f4\"}"
> |
>
> Cleaned up Define block: 
>
> |
> {{ define "opsgenie.myorg.priority"}} 
> {{ range .Alerts }}
> {{ if eq .Labels.severity "critical"}}
> P1
> {{ else }}
> P3
> {{ end }} 
> {{ end }}
> {{ end }}
> |

Just a very blind guess without having tried and without knowning
OpsGenie: Could this be whitespace-related? Your template probably
returns " <lots of spaces> P1 <lots of spaces> ", which might make it
break?

Try removing all spaces for debugging and/or adding a dash in your
template tags (such as {{- end -}}).

https://golang.org/pkg/text/template/#hdr-Text_and_spaces

(In addition to what Simon said about multiple alerts)

Kind regards,
Christian

[Romenyrr]

@Christian - great callout on the whitespaces, it worked! I tried both ways, by just putting everything on one line and the second way by updating my formatted code and adding the `-` to each tag

@Simon - as you predicted ranging through the alerts when I have multiple ones broke the reciever. I was trying to add logic to simplify things, but that didn't work. So I'm guessing this is a perfect use case for notification templates? I can use one template for P1 alerts, and another for P3 and so on. Am I on the right track here? 

[Christian Hoffmann]

Hi,

On 1/4/20 9:49 AM, Romenyrr wrote:
> @Christian - great callout on the whitespaces, it worked! I tried both
> ways, by just putting everything on one line and the second way by
> updating my formatted code and adding the `-` to each tag

Glad to hear.

> @Simon - as you predicted ranging through the alerts when I have
> multiple ones broke the reciever. I was trying to add logic to simplify
> things, but that didn't work. So I'm guessing this is a perfect use case
> for notification templates? I can use one template for P1 alerts, and
> another for P3 and so on. Am I on the right track here? 

Not sure if notification templates are really the solution. If you want
to assign alerts with a specific severity to a specific OpsGenie
priority, then your best bet is probably adding the severity label to
the group_by list.
That way, Alertmanager will only group alerts whose priority is identical.
You should also refer to the label via {{ .GroupLabels.severity }} then.

https://prometheus.io/docs/alerting/configuration/#route

Kind regards,
Christian