Alertmanager: how to split alerts in email notifications

[anya...@gmail.com]

Hi, colleagues!

I setup and configured Alertmanager for getting email notifications, but all of my alerts about different environments are gathered in one email and sorted by alphabetical order by job_name. In test environment there are approx. 250 alerts in one email. It doesn't work for urgents reaction. 

Prometheus rules have 3 rules per 1 env. Now I'm testing 7 envs.

Prometheus config:

rule_files:

    - "ft4rules.yml"

Alertmanager config:

receivers:

- name: 'EMails'

  email_configs:

    - to: 'x...@xxx.com

      from: 'z...@zzz.com'

Rules.yml:

groups:

- name: FT4

  rules:

  - alert: Instance Code 0 Alert

    expr: probe_http_status_code {job="FT4"} < 200

    for: 1m

    labels:

      severity: "critical"

    annotations:

      title: "Instance Status"

      description: "Store {{ $labels.instance }} is down. Error Code is {{ $value }}"

# Alert for instances http code less 200      

  - alert: Instance Health Down Alert

    expr: probe_http_status_code {job="FT4"} > 399

    for: 1m

    labels:

      severity: "critical"

    annotations:

      title: "Instance Status"

      description: "Store {{ $labels.instance }} is down. Error Code is {{ $value }}"

Could you help me to split it?

Thank you in advance!

[Stuart Clark]

You want to look at the group options.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

[anya...@gmail.com]

Config is very usual.

global:

  resolve_timeout: 5m

 # The smarthost and SMTP sender used for mail notifications.

  smtp_smarthost: 'smtphost:25'

  smtp_from: 'z...@zzz.com'

  smtp_auth_username: 'z...@zzz.com'

  smtp_auth_password: ''

  smtp_require_tls: false

  

  

# The directory from which notification templates are read.

templates: 

    - 'C:\AlertManager\templates\email.tmpl'

# The root route on which each incoming alert enters.

route:

  

  group_by: ['job_name']

  group_wait: 1m

  group_interval: 5m

  repeat_interval: 15m 

  receiver: EMails

receivers:

- name: 'EMails'

  email_configs:

    - to: 'x...@xxx.com'

      from: 'z...@zzz.com'

      send_resolved: false

четверг, 4 июня 2020 г. в 18:16:15 UTC+3, Stuart Clark:

[Harald Koch]

On Thu, Jun 4, 2020, at 11:09, anya...@gmail.com wrote:

Hi, colleagues!

I setup and configured Alertmanager for getting email notifications, but all of my alerts about different environments are gathered in one email and sorted by alphabetical order by job_name. In test environment there are approx. 250 alerts in one email. It doesn't work for urgents reaction. 

Prometheus rules have 3 rules per 1 env. Now I'm testing 7 envs.

It's not useful to have 250 alerts firing at once in testing or in a real outage. You need to find something less noisy to alert on.

As an example, if you have a fleet of 100 servers, you might want to send a single alert if more than 10 of them are down; then use a dashboard to figure out which ones (and why).

As others have mentioned you'll also want to group your alerts; at the very least separate them by environment.

--

Harald

[anya...@gmail.com]

Thank you, Harald, but it's a customer's requirement. 250 alerts are at the test process, after testing they should be much less. The main question now: how to split one email to one alert per email?

четверг, 4 июня 2020 г. в 20:16:35 UTC+3, Harald Koch:

[Christian Hoffmann]

Hi,

On 6/5/20 8:20 AM, anya...@gmail.com wrote:
> Thank you, Harald, but it's a customer's requirement. 250 alerts are at
> the test process, after testing they should be much less. The main
> question now: how to split one email to one alert per email?

Shouldn't a simple group_by: ['...'] be sufficient in your case?
https://prometheus.io/docs/alerting/configuration/#configuration-file

Kind regards,
Christian

[anya...@gmail.com]

Hi, Christian!

Thank you for your answer, but could you a little bit explain your point? I don't understand it. I used 'group_by' in my config. If I understood you correctly, you offered to comment the 'group_by' block, don't you?

пятница, 5 июня 2020 г. в 09:25:39 UTC+3, Christian Hoffmann:

[Stuart Clark]

On 05/06/2020 08:23, anya...@gmail.com wrote:

Hi, Christian!

Thank you for your answer, but could you a little bit explain your point? I don't understand it. I used 'group_by' in my config. If I understood you correctly, you offered to comment the 'group_by' block, don't you?

The suggestion is to use the exact string Christian gave, with the explanation being in the documentation he linked to.

пятница, 5 июня 2020 г. в 09:25:39 UTC+3, Christian Hoffmann:

Hi,

On 6/5/20 8:20 AM, anya...@gmail.com wrote:
> Thank you, Harald, but it's a customer's requirement. 250 alerts are at
> the test process, after testing they should be much less. The main
> question now: how to split one email to one alert per email?

Shouldn't a simple group_by: ['...'] be sufficient in your case?
https://prometheus.io/docs/alerting/configuration/#configuration-file

Kind regards,
Christian

-- 
Stuart Clark

[Christian Hoffmann]

Hi,

On 6/5/20 9:23 AM, anya...@gmail.com wrote:
> Hi, Christian!
> Thank you for your answer, but could you a little bit explain your
> point? I don't understand it. I used 'group_by' in my config. If I
> understood you correctly, you offered to comment the 'group_by' block,
> don't you?

You set group_by to ['job_name']. This tells alertmanager to batch all
alerts with the same job_name into one group. This leads to the
aggregated email you want to avoid.

I suggest changing the value of group_by. Either add additional labels
(e.g. alertname, instance as you see fit) or use the magic value ['...']
to disable the grouping altogether (i.e. letting your 250 alerts become
250 single emails).

Kind regards,
Christian

[anya...@gmail.com]

Thank you, Stuart!
I tried to change 'group_by' and comment it, but it doesn't work for me: all alerts are gathering in one email. I need to separate it, but can't.

пятница, 5 июня 2020 г. в 10:42:59 UTC+3, Stuart Clark:

[anya...@gmail.com]

Thank you a lot, Christian!
It's a kind of magic, your ['...']. I tried to comment it or stay empty [] and it doesn't work. But ['...'] works perfectly!

Thanks all of you, dear colleagues!

пятница, 5 июня 2020 г. в 10:51:51 UTC+3, Christian Hoffmann:

[Sagar]

Hi Anya, 

Can you share the snippet, I am also facing the same issue . Thanks 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/045353da-d097-4d8c-b62d-54f47220196bn%40googlegroups.com.

[anya...@gmail.com]

Hi, Sunil!

Sure! Catch:

global:

  resolve_timeout: 5m

 # The smarthost and SMTP sender used for mail notifications.

  smtp_smarthost: 'smtphost:25'

  smtp_from: 'z...@zzz.com'

  smtp_auth_username: 'z...@zzz.com'

  smtp_auth_password: ''

  smtp_require_tls: false

  

  

# The directory from which notification templates are read.

templates: 

    - 'C:\AlertManager\templates\email.tmpl'

# The root route on which each incoming alert enters.

route:

  

  group_by: ['...'] #if you want to separate all the emails. You can't combine ['...'] with any other label. only this one.

  group_by: ['job'] # if you want to group all alerts by job in one email. This label you can combine with smth else. Example: group_by: ['job', 'instance', 'error_code']

  group_by: ['job_name'] # if you want to gather all over the alerts in one email.  This label you can combine with smth else.

  group_wait: 1m

  group_interval: 5m

  repeat_interval: 15m 

  receiver: EMails

receivers:

- name: 'EMails'

  email_configs:

    - to: 'x...@xxx.com'

      from: 'z...@zzz.com'

      send_resolved: false

пятница, 5 июня 2020 г. в 11:12:41 UTC+3, sunils...@gmail.com:

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

[Sagar]

Thank you so much Anya for help. 

Thanks, 

Sunil Sagar

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/045353da-d097-4d8c-b62d-54f47220196bn%40googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/283fcf32-b165-4e89-b27b-7a95fad7695en%40googlegroups.com.