Alerts are cleared from AlertManager without sending resolve alerts notification to slack/webhook channel

[shivakumar sajjan]

Hi Everyone,

I deployed Prometheus, AlertManager as single instance on Kubernetes. I facing issue - all alerts are resolved in the group if anyone alert in the group is resolved but slack/webhook channels did receive notification for one alert but not for all fired alerts in the group .

I am  new to Prometheus/AlertManager and spent so much time for debugging but no luck. So I came here to understand why resolved alerts are not received by channels.

I have mentioned example below. Could you please help me what could be the cause for this issue?

For example:- alert1, alert2, alert3 are belonging to same group - group_by [job, instance]. Here job is engine_metrics_job and instance is 1000.

1:32 - alert2 - fired - received alerts 

 3:42 - alert1, alert2 - fired - received alerts 

3:47 - alert1 alert2, alert3 - fired - received alerts 

4:02 - alert2 - resolved - received alerts

The alert1, alert3 alerts are cleared from AlertManager without sending resolve alerts notification to slack/webhook channel. 

AlertManager build version and configuration details below

Alertmanager build version details:

Status Uptime: 2022-05-23T23:35:53.161Z Cluster Status Status: disabled Version Information Branch: HEAD BuildDate: 20191211-14:13:14 BuildUser: root@00c3106655f8 GoVersion: go1.13.5 Revision: f74be0400a6243d10bb53812d6fa408ad71ff32d Version: 0.20.0

AlertManager configuration details:

global: {} route: receiver: slack-alerts-1 # Fallback. group_wait: 30s group_interval: 5m group_by: ['job', 'instance'] routes: - match: job: engine_metrics_job receiver: slack-alerts-1 continue: true - match: severity: page receiver: slack-alerts-2 continue: true - match_re: job: engine_metrics_job receiver: webhook receivers: - name: slack-alerts-1 slack_configs: - send_resolved: true username: 'Prometheus Alertmanager' channel: '#alerts-1' icon_url: '<icon_url>' api_url: '<slack_webhook-url>' text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}` *Description:* {{ .Annotations.description }} *Details:* {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}` {{ end }} {{ end }} - name: slack-alerts-2 slack_configs: - send_resolved: true username: 'Prometheus Alertmanager' channel: '#alerts-2' icon_url: '<icon_url>' api_url: '<slack_webhook-url>' text: |- {{ range .Alerts }} *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}` *Description:* {{ .Annotations.description }} *Details:* {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}` {{ end }} {{ end }} - name: webhook webhook_configs: - send_resolved: true url: 'https://localhost:8180/service/event/webhook'

Thank you,

Shiva

[shivakumar sajjan]

I added AlertManager Build version and configuration details:

AlertManager Build version: