Getting rate-limited errors on slack webhook integration with alertmanager

204 views
Skip to first unread message

Seema Phalke

unread,
Sep 8, 2022, 1:02:35 AM9/8/22
to Prometheus Users

Getting rate-limited errors on slack integration with an alertmanager.

There was similar issue posted a while ago: https://github.com/prometheus/alertmanager/issues/2111
but looks like it should be fixed in 0.24.0 version.

Should I be ignoring these errors or would need to investigate further?

Environment
System information:

Docker container

Alertmanager version:

v0.24.0

Prometheus version:

N/A

Alertmanager configuration file:
N/A - contains confidential information

Prometheus configuration:

 - name: slack-notifications-abc-default
    slack_configs:
      - api_url: https://hooks.slack.com/services/{}
        channel: alerts-abc-services
        send_resolved: true

Logs:

ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.455Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
Reply all
Reply to author
Forward
0 new messages