alertmanager delays in sending slack alerts

[Robert Morman]

Good morning!

I'm new to prometheus and wondering what is causing a delay in receiving resolved slack alerts.  I enabled debug logging and have tried changing resolve_timeout, yet there is still a delay beyond what the configuration is set.  I receive the initial InstanceDown slack alert quickly and when expected, but you can see from the debug logs that prometheus is sending a resolved alert to alertmanager, but alertmanager is delaying flushing and sending to slack.  I tried peering into the alertmanager data, but couldn't tell if prometheus is sending EndsAt.  resolve_timeout seems to have no effect.  It always seems to send 5m after the active alert is triggered.

######### alertmanager version #########

root@alertmanager1:/etc/alertmanager# alertmanager --version
alertmanager, version 0.20.0 (branch: HEAD, revision: f74be0400a6243d10bb53812d6fa408ad71ff32d)
  build user:       root@00c3106655f8
  build date:       20191211-14:13:14
  go version:       go1.13.5

######### alertmanager.yml #########

global:
  resolve_timeout: 15s

templates:
- '/etc/alertmanager/*.tmpl'

route:
  repeat_interval: 1h
  receiver: critical

receivers:
- name: 'critical'
  slack_configs:
  - api_url: https://hooks.slack.com/services/XXXXXXX/XXXXXXXX/XXXXXXXX
    channel: '#alerts'
    send_resolved: true
    title: '{{ template "title" . }}'
    text: '{{ template "slack_message" . }}'

######### prometheus alerts config #########
groups:
- name: example
  rules:
  # Alert for any instance that is unreachable
  - alert: InstanceDown
    expr: up == 0
    labels:
      severity: page
    annotations:
      summary: "Instance {{$labels.instance}} down"
      description: "{{$labels.instance}} is down!"

######### alertmanager debug logs #########
Apr 16 13:13:06 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:13:06.458Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=InstanceDown[bd6501b][active]
Apr 16 13:13:06 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:13:06.458Z caller=dispatch.go:465 component=dispatcher aggrGroup={}:{} msg=flushing alerts=[InstanceDown[bd6501b][active]]
Apr 16 13:14:36 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:14:36.456Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=InstanceDown[bd6501b][active]
Apr 16 13:15:36 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:15:36.456Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=InstanceDown[bd6501b][resolved]
Apr 16 13:17:06 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:17:06.455Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=InstanceDown[bd6501b][resolved]
Apr 16 13:18:06 alertmanager1-mon-prd-cle alertmanager[28061]: level=debug ts=2020-04-16T13:18:06.458Z caller=dispatch.go:465 component=dispatcher aggrGroup={}:{} msg=flushing alerts=[InstanceDown[bd6501b][resolved]]

[Julius Volz]

Yeah, resolution notices (as well as any other changes to an alert group) can take up to "group_interval" time to be delivered, which defaults to 5m: https://github.com/prometheus/alertmanager/blob/04ca50712575584666754d2245f70a66e2009ccd/dispatch/route.go#L33

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/1291a5cb-ed7e-45ff-aa69-2430dbec0208%40googlegroups.com.

[Robert Morman]

oh nice, i read and missed that.  many thanks Julius; i will try changing that setting.