Alertmanager Queue Issue.

[yagyans...@gmail.com]

Hi. I have around 80 alert rules configured. These days I have started receiving warning from Alertmanager saying "component=cluster msg="dropping messages because too many are queued" current=4130 limit=4096".

What exactly does this mean? Are 4130 alerts in the pending queue? How can so many alerts generate? I have currently 430 alerts(Warning + Critical). How are these many alerts queued? And what to do to overcome this issue?

Below are my global alertmanager configurations.

global:
  resolve_timeout: 5m
  slack_api_url: "blahblah"
route:
  group_by: ['cluster']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'slack-channel'

Also, out of 80 alert rules, some of them are for warning, for which I don't need an alert notification but I need them for my visualization and dash-boarding purposes. Is there any way to disable some of the alert-rules to not even go to the global receiver?

Can someone please help here? Some of my critical alerts are getting blocked.

Thanks in advance.