Alertmanager email issues

[Daniello]

Hi everyone
I am running kube-prometheus on K8s cluster - installed with helm.
Additionally I set up: AlertManagerConfig and PrometheusRules CRD

Issue:

I have two set of alerts, one of them called PodErrorAlert and the other PodPendingFor30mAlert. When I check the UI i can see both of the alerts, they get triggered. However only PodPendingFor30mAlert succesfully sends email. 
I do see a difference on how the alerts are being treated in logs but not sure how to solve that issue

Technical info/ logs/ manifests

1. You can find alertmanager debug logs here

https://pastebin.com/uqyUAW6N 

2. AlertmanagerConfig

https://pastebin.com/YxVtF4Uh

3. PrometheusRules

https://pastebin.com/s12DugCQ

[Daniello]

Any help is greatly appreciated :) 

[Brian Candler]

> I am running kube-prometheus on K8s cluster - installed with helm.

Software versions?

> When I check the UI

Which UI are you referring to? Prometheus' web UI has an "alerts" section, and Alertmanager's web UI also has an "alerts" section. Do they both show the active PodErrorAlert?

Check the exact set of labels on each active alert, to see if there's any significant difference. AFAICS they should both have namespace, pod and severity.

Does the PodErrorAlert alert remain active for the full groupWait period (30 seconds), or does it keep resolving and firing again? If the alerts are resolving too quickly, you could try this:

        - alert: PodErrorAlert
          for: 3m
          keep_firing_for: 3m    # (Prometheus v2.42.0+)