Using inhibit_rules to silence alert from certain URIs

[Ashwini Thillai]

I have got an alert to report slow HTTP Endpoints on our system. I would like to silence an alert with inhibit_rules for certain URIs using the label filters but alertmanager seems to error this out as an invalid label name. 

My usecase is to keep the alert firing but silence only when below condition ( uri label) is met.  i'd appreciate any leads/support. Thank you

For Example:

inhibit_rules:

- target_match:

uri: '/work/info'

source_match:

alertname: 'Slow HTTP Endpoints '

[Brian Candler]

What you need is to use routing rules to route the alerts which are not of interest to a null receiver.  Something like this (untested):

route:

  receiver: my_normal_receiver

  routes:
    - receiver: null
      match:
        uri: '/work/info'

receivers:

  - name: null

  - name: my_normal_receiver

    email_configs:

      - ... etc

The routing tree will first process all the rules in the 'routes' section, and if none of them match, it falls back to the sibling 'receiver' value. You can test them using the tree viewer at https://prometheus.io/webtools/alerting/routing-tree-editor/

(Inhibit rules are meant for something else - when alert X is firing, then silence alert Y - and there must be some common labels to correlate them with an "equal" rule. See the documentation here and here and the example)

[Ashwini Thillai]

Thank you so much  , it worked.