Duplicate alert notification seen with Alertmanager configured as Netcool

[Harshita Jaiswal]

Hello All,

We are using a highly available Prometheus setup by using Prometheus Operator as our driving agent. We have 2 Prometheus instances running along with 2 Alertmanagers configured to receive Alarms. 
The issue we are observing is, Alertmanager is sending duplicate Clear alert notifications for certain Alarms which is creating discrepancies at the end user. We are using Netcool as our Webhook receiver for Alerts notification.

We are using kube-prometheus-stack helm chart to deploy our Prometheus and alertmanager instances and other components 
Kube-prometheus-stack helm chart Version: "18.0.1" K8s Version: "v1.20.7"
Below is our Alertmanager configuration used with Netcool:
global: resolve_timeout: 5m receivers: - name: default-receiver webhook_configs: - url: http://<ip>:port/probe/webhook/prometheus send_resolved: true - name: Netcool-Receiver webhook_configs: - url: http://<ip>:port/probe/webhook/prometheus send_resolved: true route: group_interval: 5m group_wait: 30s receiver: default-receiver repeat_interval: 3h group_by: [ 'alertname' ] routes: - receiver: default-receiver group_wait: 30s group_interval: 5m repeat_interval: 3h continue: true - receiver: Netcool-Receiver group_wait: 10s group_interval: 5m repeat_interval: 3h continue: true templates: - /etc/alertmanager/config/*.tmpl

Below is our Prometheus ecosystem:

[cloud-user@incredibles1-bastion-1 ~]$ kubectl get pods -n occne-infra | grep kube
alertmanager-occne-kube-prom-stack-kube-alertmanager-0           2/2     Running     0          11d
alertmanager-occne-kube-prom-stack-kube-alertmanager-1           2/2     Running     65         90d
occne-kube-prom-stack-grafana-78d94b6868-kwpwz                   2/2     Running     2          90d
occne-kube-prom-stack-kube-operator-677767c6d8-s4j67             1/1     Running     2          11d
occne-kube-prom-stack-kube-state-metrics-85967b9588-csrtj        1/1     Running     2          90d
occne-kube-prom-stack-prometheus-node-exporter-47n4q             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-5f8m2             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-5rvsr             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-7hs57             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-7nv7d             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-7vffn             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-8rg7c             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-kz62p             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-l2rsz             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-p662j             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-prkhl             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-q5fnf             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-r7mjv             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-t4wcm             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-vq25w             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-vz9qn             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-xlb8c             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-xnwjr             1/1     Running     1          90d
occne-kube-prom-stack-prometheus-node-exporter-z564n             1/1     Running     1          90d
prometheus-occne-kube-prom-stack-kube-prometheus-0               2/2     Running     3          90d
prometheus-occne-kube-prom-stack-kube-prometheus-1               2/2     Running     3          90d
[cloud-user@incredibles1-bastion-1 ~]$

Please let me know if anymore info is required I will attach. 
Thank you for your time and help.