Alert Segregation

[Adewunmi Ademule]

Hello,

I have configured my prometheus.yml and alertmanager.yml file to look like below but I need help with making alert manager send email alerts to different users per instance(i.e email alerts for instance A1 goes to only it...@xyz.com and  instance B2 goes only to de...@xyz.com). I currently have all email alert sent to  de...@xyz.com and  it...@xyz.com at the same time which seem too noisy at times.

  prometheus.yml   

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093
        - localhost:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
    - "alert_rules.yml"
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['A1:9182','B2:9182'']
   
  - job_name: "ssl"
    metrics_path: /probe
    static_configs:
      - targets:
          - A1:1667
          - B2:1667

alertmanager.yml

route:
  group_by: [Alertname]
  receiver: email
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 10m

receivers:
- name: email
  email_configs:
  - to: it...@xyz.com, de...@xyz.com
    from: a...@xyz.com

Any suggestion would be highly appreciated.

Regards,

[Laurent C.]

Hi,

I think https://www.prometheus.io/docs/alerting/latest/configuration/#example should help.

If you don't want to receive emails to the 2 email addresses at the same time, you have to define 2 different receivers:

receivers:
- name: email-it
  email_configs:
  - to: it...@xyz.com
    from: a...@xyz.com

- name: email-de
  email_configs:
  - to: de...@xyz.com
    from: a...@xyz.com

And then, in section "route:" > "routes:", you will have some blocks including "match_re:" to route alerts matching some labels to the appropriate receiver, so something like:

- match_re:

  instance: A1

  receiver: email-it

- match_re:

  instance: A2

  receiver: email-de

[Ade]

Sorry it took so long to get back to this but wanted to update the group incase anyone else is facing this same issue. The below worked for me:

 

routes:
  - match_re:
      instance: ~"A1,B2"  #This is to include email-it in all alerts for multiple instances A1 and B2
      severity: critical|warning|high #This picks all severity i.e critical or warning or high
    receiver:  email-it
  - match_re:
      instance: A2 # This is to include email-de in alert for only instance A2
      severity: critical|warning|high
    receiver: email-de

Thanks,

Ade

[Brian Candler]

Thanks for the update.  A couple of minor notes:

 - match_re:
      instance: ~"A1,B2"  #This is to include email-it in all alerts for multiple instances A1 and B2

Are you sure that's right?  I think it should be

 - match_re:
      instance: A1|B2

Note that if you want to include alerts of *all* severity, then you don't need the "severity: critical|warning|high" condition.  Just leave it out.  Then you will match all alerts regardless of severity (even no severity set).

Incidentally, there's a new matcher syntax which is more like normal PromQL label matching:

 - matchers:
     - 'instance =~ "A1|B2"'

match and match_re are deprecated in favour of this, although I don't expect they'll be removed for a long time, if ever.

[Ade]

Thanks for the update Brian. I tried your suggestion but that didn't work as alertmanager failed to start and I had to revert to the working state. The version of my alertmanager is 
Build 2.18.1 and thinking this might be the problem. Any ideas??

[Brian Candler]

You appear to be looking at the wrong version number.  I'm running alertmanager 0.23.0; there is no 1.x or 2.x release.  It looks like route matchers were added in alertmanager 0.22.0 (May 2021).

You could be running prometheus 2.18.1 though - which is fairly old, released in May 2020.

[Adewunmi Ademule]

You are right. I was checking the Prometheus build instead. My alertmanager is version 0.21.0

Thanks!

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/YosD_lC1MW4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/adc58bf3-cbc0-4a81-9e11-bbb9455e38bbn%40googlegroups.com.