Getting rate-limited errors on slack integration with an alertmanager.
but looks like it should be fixed in 0.24.0 version.
Environment
System information:
Docker container
Alertmanager version:
v0.24.0
Prometheus version:
N/A
Alertmanager configuration file:
N/A - contains confidential information
Prometheus configuration:
- name: slack-notifications-abc-default
slack_configs:
- api_url:
https://hooks.slack.com/services/{} channel: alerts-abc-services
send_resolved: true
Logs:
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.455Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"