Alerts fired from Prometheus are missing on PagerDuty

[amar]

Hello All,

We have executed the below query on Prometheus UI and could see multiple alerts gets fired.

ALERTS{alertname="ProdAlert", alertstate="firing"}

The alerts are getting routed properly to our Slack and PagerDuty receivers but still some alerts are missed even though the above expression confirms that Prometheus has actually fired the alerts based on the alerting rules.

It looks like some alerts are getting unprocessed from alertmanager.

alertmanager_alerts{state="unprocessed"}

We looked at the alertmanager log and there is nothing relevant that explains the issue. May be we should enabled debug logging and check further.

Is there anything we can check to identify the issue here?

Please let me know.

Thanks,

Amar