Resuable Template not found

[Mark Leone]

I'm following this example to define a re-usable template: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/

and alertmanager is not able to find the template

This is my alertmanager config file:

global:
    smtp_smarthost: ''my.host:25'
    smtp_hello: 'my.system'
    smtp_from: 'my.adderss'

route:
    receiver: test-email-receiver
    group_by: [label1, label2]

receivers:

• name: 'test-email-receiver'
  email_configs:
  • to: 'som...@example.org'
    text : '{{ template "custom.email" . }}'

templates:

• '/etc/alertmanager/templates/custom-email-template.tmpl'

This is the contents of /etc/alertmanager/custom-email-template.tmpl:

{{ define "custom.email" }}Test{{ end }}

I'm running the alertmanager:latest docker image, and it fails with the error: template "custom.email" not defined

The custom template file is written to the local docker volume by a nomad template stanza in the docker driver config, and mapped to /etc/alertmanager/templates/custom-email-template.tmpl. I removed the text entry in email_configs so I can successfully start the container and connect to it with docker exec. I verified that the custom template file is where I expect it to be and it has the expected contents. Alertmanager just refuses to recognize the template. Either the example is wrong, or I'm missing something.

[Mark Leone]

I still haven't figured out why it's not working. Developers responding to the GitHub issue I opened believe I've got something configured wrong, since no one else has reported this problem. Can someone confirm that they're able to use custom templates per the instructions referenced in the OP? If so, is there something else not mentioned in the instructions that I need to do?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f0f16572-6bf7-4085-b9a7-d5948e3b41f3%40googlegroups.com.

[Julien Pivotto]

Which version is this? Alertmanager should start with an unknown template.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAFHj5BddEFUdf_dd-3o9gvi65kN-LcYjEn6Ee0P_YieQ%2BHns7Q%40mail.gmail.com.

[Mark Leone]

Here is my alertmanager version info:

alertmanager, version 0.20.0 (branch: HEAD, revision: f74be04)
build user: root@00c3106655f8
biold date: 20191211-14:13:14
go version: gol.13.5

Please clarify what you mean when you say "Alertmanager should start with an unknown template. ". I reference a template in the text field of the email_configs setting. If I reference the default template (either explicitly or by not specifying the text property), it works. It finds the template and sends a templated message as expected.If I reference the custom template that I defined (custom.email), alertmanager fails to start and says it can't find the template I referenced. AFAICAT I followed the example exactly in defining a custom template and pointing to it in the "templates" config setting. I also exec into the docker image and verify that the custom template file is where I pointed alertmanager to, permissions are opened, and it contains the text I expect it to contain.I did see a report somewhere that you can't override the default templates, so I gave my template a different name; but still alertmanager is not able to find it.

[Brian Candler]

The configuration file that you posted was mangled - partly by your E-mail client I guess, and partly by you (there are two single quotes before ''my.host')

However, if I use exactly the following configuration file:

global:

  smtp_smarthost: 'my.host:25'

  smtp_hello: 'my.system'

  smtp_from: 'my.adderss'

route:

  receiver: test-email-receiver

  group_by: [label1, label2]

receivers:

  - name: 'test-email-receiver'

    email_configs:

      - to: 'som...@example.org'

        text: '{{ template "custom.email" . }}'

templates:

  - '/etc/alertmanager/templates/custom-email-template.tmpl'

then alertmanager starts just fine:

root@prometheus:~# /opt/alertmanager/alertmanager --config.file=/tmp/alert.yml

level=info ts=2020-06-06T17:16:00.765Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)"

level=info ts=2020-06-06T17:16:00.766Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)"

level=info ts=2020-06-06T17:16:00.770Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=10.12.255.33 port=9094

level=info ts=2020-06-06T17:16:00.777Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s

level=info ts=2020-06-06T17:16:00.871Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/tmp/alert.yml

level=info ts=2020-06-06T17:16:00.875Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/tmp/alert.yml

level=info ts=2020-06-06T17:16:00.888Z caller=main.go:497 msg=Listening address=:9093

level=info ts=2020-06-06T17:16:02.777Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00028059s

level=info ts=2020-06-06T17:16:10.778Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.001491515s

q
^C
level=info ts=2020-06-06T17:16:40.881Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..."

root@prometheus:~# /opt/alertmanager/alertmanager --version

alertmanager, version 0.20.0 (branch: HEAD, revision: f74be0400a6243d10bb53812d6fa408ad71ff32d)

  build user:       root@00c3106655f8

  build date:       20191211-14:13:14

  go version:       go1.13.5

This is despite the fact that I haven't even created /etc/alertmanager/templates/custom-email-template.tmpl

Therefore, I can only surmise that the configuration you're testing with is not the one you posted. Maybe you haven't given the correct command-line argument to alertmanager to tell it which config file to read, and so it's reading a different one than the one you think it is.

[Mark Leone]

Brian, thanks for taking the time to run my config. The error you noted was just in my email. I had to manually enter the config file because I'm running alertmanager on an isolated network.

I finally discovered the problem. I'm running alertmanager in a docker container, and I'm using nomad to run that container. To get the alertmanager config file and the custom template file into the docker container, I use a nomad template stanza to copy Here Document text to the local docker volume, and I map that volume to the location where alertmanager expects the files to be. After discovering that I don't get the error when I start the docker container manually, only when I run it in nomad, I realized that the "template not defined" error was referring to the nomad template, not the alertmanager go template. They both use {{ and }} for template delimiters, but nomad allows you to specify an alternate character.

So my nomad template stanza now looks like this:

template: {
   change_mode = "noop"

   destination = "local/email-tmepolate.tmpl"

   left_delimiter = "[["

   right_delimiter = "]]"

   data = <<EOH

---

<alertmanager config content here, including the text text: '{{ template "custom.email" . }}'>

EOH

               }

And that works. Without the left_delimiter and right_delimeter overrides, nomad fails the job while parsing the template stanza, thinking that I'm referring to a nomad template named "custom.email", which of course does not exist. So the alertmanager was never even started, as nomad failed the job when parsing its template stanza.

I was completely on the wrong track by not realizing it was the nomad template that was having the problem. Thanks, everyone, for your help, especially as the problem had nothing to do with alertmanager.

-Mark

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9de8ab3d-7f4e-43e1-be5c-665908e776cdo%40googlegroups.com.