How should configue the alertmanager alert interval about the alert notifications of resolved

[zhang windy]

Hi, buddies, I met a problem,  while the alert in it had a listed end time of 2020-03-11T09:59:24, the actual webhook was only sent at 2020-03-11T10:13, it looks like 15 minutes interval, I want to decrease the period time to 5s, so I carefully look for the configuration about alertmanager, but cannot found anything about it, how can I set it?  thank you very much.

below is configuration of alertmanger:

 

"global": 

  "resolve_timeout": "5m"

  smtp_smarthost: 'XXXXXXX'

  smtp_from: 'XXXXXXX'

  smtp_auth_username: 'XXXXXXX

  smtp_auth_password: 'XXXXXXX'

  smtp_require_tls: false

"receivers": 

- "name": "mail"

  email_configs:

  - to: '7935...@qq.com'

- "name": "webhook"

  webhook_configs:

  - url: 'http://flask-alert-service.monitoring:5000/send'

  #- url: 'http://flask-alert-service-2.monitoring:5000/send'

  - url: 'http://aipaas-alert-service.aipaas:10019/alert/message'

  - url: 'http://aipaas-serving-model-service.aipaas:10014/serving/alert/message'

"route": 

  "group_by": ['alertname', 'cluster','service'] 

  #- "job"

  "group_interval": "1h"

  "group_wait": "30s"

  "receiver": "webhook"

  "repeat_interval": "12h"

  "routes": 

  - "match": 

      "alertname": "DeadMansSwitch"

    "receiver": "webhook"

  - "match":

      severity: warning

    "receiver": "webhook"

[zhang windy]

maybe I found some usefull information in alertmanger debug logs, it shows sometime the resolved alert messages didn't send to webhook , I feel a little confused, could you give me some ideas? thank you.

for details, see below：

level=debug ts=2020-03-11T09:08:58.366265467Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][active]

level=debug ts=2020-03-11T09:09:24.672978327Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:09:58.365912476Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:11:24.675325088Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:11:58.365464319Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:13:24.673909146Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:13:58.36588506Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:15:24.678054827Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:15:58.366574391Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:17:24.674620753Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:17:58.365941732Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:19:24.673374598Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:19:58.370509047Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:21:24.675464534Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:21:58.366838423Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:23:24.674355197Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:23:58.366254181Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[65e7146][resolved]

level=debug ts=2020-03-11T09:58:24.674017138Z caller=dispatch.go:104 component=dispatcher msg="Received alert" alert=IstioService_429_Error[4a3874a][active]

level=debug ts=2020-03-11T09:58:54.67426069Z caller=dispatch.go:343 component=dispatcher aggrGroup="{}:{alertname=\"IstioService_429_Error\"}" msg=flushing alerts=[IstioService_429_Error[4a3874a][active]]

level=debug ts=2020-03-11T09:58:54.814196814Z caller=nflog.go:538 component=nflog msg="gossiping new entry" entry="entry:<group_key:\"{}:{alertname=\\\"IstioService_429_Error\\\"}\" receiver:<group_name:\"webhook\" integration:\"webhook\" idx:2 > timestamp:<seconds:1583920734 nanos:772370937 > firing_alerts:15719501330978136231 > expires_at:<seconds:1584352734 nanos:772370937 > "

在 2020年3月11日星期三 UTC+8下午8:45:03，zhang windy写道：

[Simon Pasquier]

Alertmanager evaluates alert groups at the given group_interval value
(1h in your case) which would explain the "long" delay. You need to
decrease this value to get notifications earlier. I'd recommend not
less than 1 minute though.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d5d8541c-3b25-4acf-96a1-bb8c58aa9a3f%40googlegroups.com.