1st service down alert repeating when 2nd service down after few minutes

[Sandeep Rao Kokkirala]

Hi Team,

I am encountering below issue. Appreciate if any one responds. 

consider 1st service is down ..our alertmanager is triggers  the alert ...when 2nd service is down after 10 minutes ..that time it triggering both 1st service and 2nd service alerts ...1st service alert already triggered so  we don't want 1st service alert to repeat.

Below is alert manager configuration file:

global:

  resolve_timeout: 30s

route:

  receiver: 'email_receiver_dummy'

  group_wait: 360h

  group_interval: 480h

  repeat_interval: 480h

  group_by: ['alertname']

  routes:

  - receiver: 'email_receiver'

    match_re:

      alertname: mssql-cluster-down|NodedownAlert|ServiceDown|Service-Down|mssql-secondary-node-down|mssql-primary-node-down|Mongoservicedegraded|Mongoservicedown|gfsdown|gfs-master1-down|gfs-master2-down|gfs-master3-down|Servicedegrade|Service-Degrade

    group_wait: 30s

    group_interval: 5m

    repeat_interval: 24h

    group_by: ['alertname']

    continue: true

  - receiver: 'webhook_receiver'

    match_re:

      alertname: mssql-cluster-down|NodedownAlert|ServiceDown|Service-Down|mssql-secondary-node-down|mssql-primary-node-down|Mongoservicedegraded|Mongoservicedown|gfsdown|gfs-master1-down|gfs-master2-down|gfs-master3-down|Servicedegrade|Service-Degrade

    group_wait: 30s

    group_interval: 5m

    repeat_interval: 24h

    group_by: ['alertname']

receivers:

  - name: email_receiver_dummy

    email_configs:

      - smarthost: xxxxxx:25

        auth_username: xxxxx

        auth_password: yyyyyy

        require_tls: true

        to: xxxxxxxxxx...@test.com

        from: xx...@yyyyyyy.org

        html: '{{ template "__email_text_list" . }}'

        headers:

          subject: '{{ template "_email_subject" . }}'

  - name: email_receiver

    email_configs:

      - smarthost: xxxxxxxxxxxxxx:25

        auth_username: xxxxxxxxxxxxx

        auth_password: xxxxxxxxxxxxxxxxxx

        require_tls: true

        to: xxxxxxxxxxxxxxxxxxxx@org

#        to: xxxxxxxxxxxxxxxxxxxx@org

        from: xxxxxxxxxxxxxxxxxxxx@org

        html: '{{ template "__email_text_list" . }}'

        headers:

          subject: '{{ template "_email_subject" . }}'

  - name: webhook_receiver

    webhook_configs:

      - send_resolved: false

        url: http://prometheus-webhook.ns-prometheus-mgmt:8080/v1/webhook?group=operation

templates:

  - '/etc/alertmanager/config/notification.tmpl'

[Brian Candler]

On Monday, 14 September 2020 11:38:43 UTC+1, Sandeep Rao Kokkirala wrote:

consider 1st service is down ..our alertmanager is triggers  the alert ...when 2nd service is down after 10 minutes ..that time it triggering both 1st service and 2nd service alerts ...1st service alert already triggered so  we don't want 1st service alert to repeat.

It's sending a new *grouped* alert which contains both the original alert and the 2nd alert, grouped together.  You asked for this with   group_by: ['alertname']   which means that two alerts with the same 'alertname' label should be considered part of the same group.

You can disable grouping entirely with:    group_by: ['...']  (yes, that's literally three dots in there)

https://prometheus.io/docs/alerting/latest/configuration/#route

[Sandeep Rao Kokkirala]

Thanks  Brian . it's working  

[Sandeep Rao Kokkirala]

HI Brian & Team,

Alertmanager federation issue. 

MGMT zone which ever the alerts we trigger it triggering the alerts as expected i.e repeat interval every 24 hours but the alerts which are coming from appzone it is not working as expected. 

Could you please help any way i can add configuratoin from app zone to trigger the alerts same like mgmt zone. 

Appreciate your reply on this. 

Below is my configuration files:

Alertmanager configuration MGMT zone:

App zone prometheus configuration where we have configured federation which sends the alerts app to mgmt tier in our environment. 

prometheus:

  service:

    nodePort: 31540

    type: NodePort

    sessionAffinity: ClientIP

  prometheusSpec:

    image:

      repository: quay.io/prometheus/prometheus

      tag: v2.15.2

    externalLabels:

      cluster: 'qa-app'

    externalUrl: "http://prometheus-operator-prometheus.ns-prometheus-qa:9090/"

    secrets:

      - etcd-client-cert

    replicas: 3

    storageSpec:

       volumeClaimTemplate:

         spec:

           storageClassName: nfs-client

           accessModes: ["ReadWriteOnce"]

           resources:

             requests:

               storage: 150Gi

    additionalAlertManagerConfigs:

      - static_configs:

        - targets: [ 'mgmtzoneip:32541' ]

        scheme: https

        tls_config:

          insecure_skip_verify: true

  

  

MGMT zone which ever the alerts we trigger it triggering the alerts as expected i.e repeat interval every 24 hours but the alerts which are coming from appzone it is not working as expected. 

Could you please help any way i can add configuratoin from app zone to trigger the alerts same like mgmt zone. 

Appreciate your reply on this.