Prometheus alerts to use local timezone

[bgz]

I am trying to get alerts with a local timezone as a timestamp and i don't seem to be getting ti right all notifications go out as UTC.. I want to set this in two places in annotations and in my template. But these are just ignored by prometheus. I found this link:

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusAlertTimeFormatting

But the link does not explain where do you need to set this exactly. 

The way i set it in my email template is like this in the alertmanager.yml file:

text: >-

        {{ range .Alerts -}}

        *Time: {{ .StartsAt.Local.Format "Sat Aug 14 10:15:40 SAST 2021" }} {{ end }}

        *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

        *Description:* {{ .Annotations.description }}

        *Details:*

          {{ range .Labels.SortedPairs }}: *{{ .Name }}:* `{{ .Value }}`

          {{ end }}

        {{ end }}

I tried annotations in the alerts file as well but these are just flat out ignored.

 annotations:

      timestamp: >

         time:  {{ range .Alerts.Firing }}

                      {{ .StartsAt.Local.Format "Sat Aug 14 10:15:40 SAST 2021" }}

                {{ end }}

What am i doing wrong? Has anyone gotten it to work? I am using alertmanager 0.21.0 and Prometheus 2.24.1

[Brian Candler]

Just a thought, but what local timezone is your system set to?  (e.g. Ubuntu/Debian: "dpkg-reconfigure tzdata")

[bgz]

My timezone is set to SAST on the server but all alerts are stuck in UTC.

[Bjoern Rabenstein]

On 15.08.21 01:44, bgz wrote:
> I am trying to get alerts with a local timezone as a timestamp and i don't
> seem to be getting ti right all notifications go out as UTC.. I want to set
> this in two places in annotations and in my template. But these are just
> ignored by prometheus. I found this link:
>
> https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusAlertTimeFormatting
>
> But the link does not explain where do you need to set this exactly.

The description there is just for alert templates (as evaluated on the
Alertmanager), so they only affect what you send out to receivers
(Slack, Pagerduty, ...). Alertmanager doesn't use those templates for
its own UI.

> The way i set it in my email template is like this in the alertmanager.yml
> file:
>

> *text*: >-
>
> {{ range .Alerts -}}
>
> **Time*: {{ .StartsAt.Local.Format "Sat Aug 14 10:15:40 SAST 2021"
> }} {{ end }}
>
> **Alert*:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{
> .Labels.severity }}`{{ end }}
>
> **Description*:* {{ .Annotations.description }}
>
> **Details*:*

>
> {{ range .Labels.SortedPairs }}: *{{ .Name }}:* `{{ .Value }}`
>
> {{ end }}
>
> {{ end }}

Your formatting string has to be "15:04:05 MST 2006-01-02" (or
similar, those values, including "MST" for the timezone, are
magic). But otherwise, this seems generally to be OK. Perhaps you are
not looking at the right output? As said, the AM UI will be
unaffected. You have to look at an output that actually uses this
template to render itself. (Put a unique marker string in the template
somewhere to make sure you are actually using the template you think
you are using.)

And as Brian already said, your server needs to have the right
timezone set. You could write a small Go program and run it on your
server to see what timezone the Go runtime sees. The `main` function
simply needs the following line for that:

```Go
fmt.Println(time.Now().Local().Format("15:04:05 MST 2006-01-02"))
```

The Go `time` package even allows you to load locations from a
TZ-database, e.g. you can use the following Go code to print a time in
SAST, provided your server has a TZ-database accessible to the Go
runtime with an entry for Johannesburg:

```Go
loc, err := time.LoadLocation("Africa/Johannesburg")
if err != nil {
fmt.Println(time.Now().In(loc).Format("15:04:05 MST 2006-01-02"))
}
```

However, I'm afraid `time.LoadLocation` is not accessible in the
templating (but it might be an idea to offer it as a future
feature... note that we discussed better timezone support at the last
dev-summit, and this could be a part of it).

> I tried annotations in the alerts file as well but these are just flat out
> ignored.

Yeah, annotations don't have access to the `.StartsAt` field in the
first place.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in