Alertmanager re-firing emails every 5 days

[Nick Carlton]

Hello Everyone,

I have the following alertmanager config:

global: resolve_timeout: 23h20m http_config: follow_redirects: true enable_http2: true smtp_hello: localhost smtp_require_tls: true pagerduty_url: https://events.pagerduty.com/v2/enqueue opsgenie_api_url: https://api.opsgenie.com/ wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/ telegram_api_url: https://api.telegram.org webex_api_url: https://webexapis.com/v1/messages route: receiver: "null" group_by: - '...' continue: false routes: - receiver: alertviaemail match: notification: true continue: true group_wait: 5m group_interval: 5m repeat_interval: 30d receivers: - name: "null" - name: alertviaemail email_configs: - send_resolved: true to: xxxxxxxxxxxxxxxxxxxxx from: xxxxxxxxxxxxxxxxxxxxxx hello: localhost smarthost: smtp.office365.com:587 auth_username: xxxxxxxxxxxxxxxxxxxxxx auth_password: <secret> auth_identity: xxxxxxxxxxxxxxxxxxxxxx require_tls: true templates: - /etc/alertmanager-templates/*.tmpl

Everything works fine l. However, with this config I get email alerts every 5 days from the original fire date/time. I’ve had a read of the alertmanager documentation and cannot see what could be causing this. The alert active date stays the same in the web UI of Prometheus and Alertmanager so it’s not like the alert is resolving and then firing again. Plus I don’t get a resolved email before.I even have repeat_interval at 30 days to stop us getting re-alerted. It’s really odd that it’s specifically 5 days.

Is anyone able to advise why I get repeat alerts every 5 days?

Thanks

Nick

[Julien Pivotto]

It is because alertmanager has a default data retention of 120h, so it
does not remember sent notifications for longer.

--data.retention=120h

Regards

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1780fbea-435f-4a95-aece-2c526d015340n%40googlegroups.com.


--
Julien Pivotto
@roidelapluie