Getting rate_limited errors on slack integration with alertmanager

[Seema Phalke]

Hi,

I am getting rate_limited errors on slack integration with the alertmanager.

There was a similar issue https://github.com/prometheus/alertmanager/issues/2111 reported in the past which should be fixed on 0.24.0 version I am using.

Should I be ignoring these errors as alerts are still pouring in, or do I need to investigate this further?
What is the best way to troubleshoot to see if alerts are missed sending to the slack channel?

Environment

System information:

Docker container

Alertmanager version:

v0.24.0

Prometheus version:

N/A

Alertmanager configuration file:
N/A - contains confidential information

Prometheus configuration:

 - name: slack-notifications-abc-default
    slack_configs:
      - api_url: https://hooks.slack.com/services/
        channel: alerts-abc-services
        send_resolved: true

Logs:

ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.451Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.452Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"
ts=2022-09-07T18:42:28.455Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=2 err="slack-notifications-abc-default/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"alerts-abc-services\": unexpected status code 429: {\"retry_after\":1,\"ok\":false,\"error\":\"rate_limited\"}"

Thanks,

Seema