Why do my rules seem to fail (checking with routing tree editor)

[Danny de Waard]

When i put my config in the routing tree editor and try some routes they almost always point to default instead of the one i want.

My config:

global:
  smtp_hello: 'localhost'

route:
  group_by: [instance,severity,job]
  receiver: 'default'

  routes:

   - match:
      alertname: 'MaintenanceMode'
     receiver: 'blackhole'

   - match:
      alertname: 'TriggerWebhook'
     #receiver: 'webhook'
     receiver: 'sms-waard'

   - match:
      severity: warning
      job: PAT
     receiver: 'pat'

   - match:
      severity: warning
      job: PROD
     receiver: 'prod'
     continue: true
   - match:
      severity: warning
      job: PROD
     receiver: 'webhook'

   - match:
      severity: critical
      job: PAT
     receiver: 'pat-crit'

   - match:
      severity: critical
      job: PROD
     receiver: 'prod-crit'
     continue: true
   - match:
      severity: critical
      job: PROD
     receiver: 'sms-waard'
     continue: true
   - match:
      severity: critical
      job: PROD
     receiver: 'webhook'

   - match:
      severity: info
     receiver: 'info'

   - match:
      severity: atombomb
     receiver: 'webhook'

receivers:
  - name: 'default'
    email_configs:
     - to: 'danny.de.waard@somemaildomain' ##fill in your email
       from: 'alertmanag...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'pat'
    email_configs:
     - to: 'danny.de.waard@somemaildomain' ##fill in your email
       from: 'alertman...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'prod'
    email_configs:
     - to: 'functional.mailbox@somemaildomain, danny.de.waard@somemaildomain' ##fill in your email
       from: 'alertman...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'pat-crit'
    email_configs:
     - to: 'somenamed@somemaildomain, 55536361@somemaildomain' ##fill in your email
       from: 'critical-ale...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'prod-crit'
    email_configs:
     - to: 'functional.mailbox@somemaildomain, danny.de.waard@somemaildomain' ##fill in your email
       from: 'critical-aler...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'info'
    email_configs:
     - to: 'functional.mailbox@somemaildomain' ##fill in your email
       from: 'alertman...@superheroes.com'
       smarthost: 'localhost:25'
       require_tls: false

  - name: 'sms-waard'
    email_configs:
     - to: '55536361@somemaildomain' ##fill in your email
       from: somename@somemaildomain'
       smarthost: 'localhost:25'
       text: 'This is an example test for the sms-alert'
       require_tls: false

  - name: 'webhook'
    webhook_configs:
     - url: 'http://127.0.0.1:9000/hooks/sm9-inc'

  - name: 'blackhole'

inhibit_rules:
- source_match:
    alertname: MaintenanceMode
  target_match_re:
    severity: 'warning|critical'

If i put in the label set {job="PROD" severity="critical"} i would suspect it will route to prod-crit, sms-waardd and webhook but it falls back to default

[Brian Candler]

You need a comma:

{job="PROD",severity="critical"}

I think it should give an error for that though.  Also, it seems to ignore "continue: false/true" and just shows all the matches anyway.

[Danny de Waard]

Okay got that noob hurdle now ;)

I got another question bout this.

I have a kind of maintenance rule implemented that if a value for MaintenanceMode==1 then all alerts should be dropped. But it looks like the info alerts are not bottered by this rule?
Can anyone explain why?

(p.s. config is still the same)

Info alert is (under host rules section):

alert: loadbalancer_down_for_host expr: loadbalancer_stats < 1 for: 5m labels: severity: info annotations: description: Check the loadbalancer status. Is there a deployment?. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}. summary: The loadbalancer for {{ $labels.instance }} has been down for the last 5 minutes

Maintenance alert is (under targets rules section):

alert: MaintenanceMode expr: maintenance_mode == 1 for: 1m labels: severity: warning annotations: description: Fires during maintenance mode and is routed to a blackhole by Alertmanager summary: This is a maintenance alert

Is the section giving me troubles?

Op maandag 18 mei 2020 11:29:57 UTC+2 schreef Brian Candler:

[Stuart Clark]

On 2020-06-09 12:39, Danny de Waard wrote:
> Okay got that noob hurdle now ;)
>
> I got another question bout this.
>
> I have a kind of maintenance rule implemented that if a value for
> MaintenanceMode==1 then all alerts should be dropped. But it looks
> like the info alerts are not bottered by this rule?
> Can anyone explain why?
>
> (p.s. config is still the same)
>
> Info alert is (under host rules section):

> alert: loadbalancer_down_for_host [1] expr: loadbalancer_stats < 1 [2]

> for: 5m labels: severity: info annotations: description: Check the
> loadbalancer status. Is there a deployment?. Reported by instance {{
> $labels.instance }} of job {{ $labels.job }}. summary: The
> loadbalancer for {{ $labels.instance }} has been down for the last 5
> minutes
>
> Maintenance alert is (under targets rules section):

> alert: MaintenanceMode [3] expr: maintenance_mode == 1 [4] for: 1m

> labels: severity: warning annotations: description: Fires during
> maintenance mode and is routed to a blackhole by Alertmanager summary:
> This is a maintenance alert
>
> Is the section giving me troubles?
>
> Op maandag 18 mei 2020 11:29:57 UTC+2 schreef Brian Candler:
>
>> You need a comma:
>>
>> {job="PROD",severity="critical"}
>>
>> I think it should give an error for that though. Also, it seems to
>> ignore "continue: false/true" and just shows all the matches anyway.
>

You need to take a look at inhibit rules for this.

https://prometheus.io/docs/alerting/configuration/#inhibit_rule

--
Stuart Clark

[Danny de Waard]

I think i found my own answer... the file did change and i had allready added inhibit rules.

inhibit_rules:

- source_match:

    alertname: MaintenanceMode

  target_match_re:

    severity: 'warning|critical'

And info is not in the list ;)

Thanks for being my rubber duvk

Op dinsdag 9 juni 2020 13:52:58 UTC+2 schreef Stuart Clark: