Alert manager receiver config.

[Chrysale Tchako]

This is my current alertmanager.yml config, but all notifications sent out get sent to all emails added in the "to" section of the email configs. What changes can I make so that each receiver gets it's own individual email notification instead of grouping them together? Note the "from" email is the same for both receivers.

global:
  smtp_hello: '*******.net'

route:
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 12h
  group_by: ["alertname", "severity"]
  # Default receiver (for all alerts that don't match any specific route)
  receiver: "gmail"

routes:
  - receiver: "gmail"
    continue: true
  - receiver: "dev-receiver"

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'instance']
    routes:
      - receiver: "gmail"
        continue: true
      - receiver: "dev-receiver"

receivers:
  # Default receiver for other alerts
  - name: "gmail"
    email_configs:
      - to: "email1.net, email2.net"
        from: '*******.net'
        smarthost: 'smtp-relay.gmail.com:587'
        send_resolved: true

  - name: "dev-receiver"
    email_configs:
      - to: 'email3.net'
        from: '*******.net'
        smarthost: 'smtp-relay.gmail.com:587'
        send_resolved: true

[Brian Candler]

What version of alertmanager are you using? Your configuration is invalid, so I don't know how you're even getting alertmanager to start.

You have a top-level "routes:" block (before "inhibit_rules"), and you have another "routes:" block nested under inhibit_rules, but neither of these are allowed by Alertmanager. If I try to run your config with alertmanager-0.27.0 I get an explicit error:

ts=2025-01-02T16:41:05.905Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=amtest.conf err="yaml: unmarshal errors:\n  line 12: field routes not found in type config.plain\n  line 23: field routes not found in type config.plain"

It would be valid to nest "routes" under "route", like this:

route:
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 12h
  group_by: ["alertname", "severity"]
  # Default receiver (for all alerts that don't match any specific route)
  receiver: "gmail"

  routes:
    - receiver: "gmail"
      continue: true
    - receiver: "dev-receiver"

Note the spacing: "routes:" is indented to line up with other attributes like "receiver", not at the top level. Furthermore, if you did, it could explain the behaviour you're seeing, since this configuration requests all messages to be sent to both receivers. But since you say that's *not* your actual configuration, I think we have to start somewhere else.

How are you getting your config into alertmanager? Are you editing the alertmanager configuration directly, or is it coming via some frontend like a helm chart which spits out the config? If so, can you show the generated config, which is the actual file that alertmanager consumes?

> What changes can I make so that each receiver gets it's own individual email notification instead of grouping them together?

What do you mean by "individual email notification instead of grouping them together"?  Do you want some alerts to go to one receiver, and some alerts to go to a different receiver? If so, how do you want to choose which alerts go where?

[Chrysale Tchako]

Currently I am running alertmanager version 0.25.0, and I am making changes/modifications directly in the alertmanager.yml file, this is the file containing the information I shared in my initial inquiry. Or are referring to another file? I updated my routes config to match what you recommended but, email notifications are still getting sent to all emails in the alertmanager.yml file regardless of the receiver they are associated with. What I am trying to accomplish is a custom alert/notification for each receiver. 

Let's say I have Team A receiver for Team A resources and Team B receiver for Team B resources, I want to isolate the notifications for each team. Neither team should be aware of or get notified about the other teams resources. How can I accomplish that? What modifications should I make to my alertmanager.yml file? Also, do I need to make any modifications to my alert.rules.yml file or any other file that is?

Please let me know if you need any additional information.

[Brian Candler]

I have retested this with alertmanager 0.25.0, and it still rejects the config file you supplied as invalid, refusing to start. Therefore, I think you're *not* running with the configuration file you think you are. This is the first issue to resolve.

You can prove this to yourself by running a separate instance of alertmanager on the command line, pointing it explicitly at the config file in question:

# /opt/alertmanager-0.25.0.linux-amd64/alertmanager --config.file=amtest.conf.orig  --web.listen-address=:9193 --cluster.listen-address=0.0.0.0:9194
ts=2025-01-03T16:10:12.215Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
ts=2025-01-03T16:10:12.215Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)"
ts=2025-01-03T16:10:12.217Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=10.12.255.33 port=9194
ts=2025-01-03T16:10:12.219Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2025-01-03T16:10:12.249Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=amtest.conf.orig
ts=2025-01-03T16:10:12.249Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=amtest.conf.orig err="yaml: unmarshal errors:\n  line 12: field routes not found in type config.plain\n  line 23: field routes not found in type config.plain"
ts=2025-01-03T16:10:12.249Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=30.684346ms

(The default config file is "alertmanager.yml", but as there is no absolute path, it will be in whatever the current working directory is when alertmanager is started. This may not be where you think it is, and in particular, is not necessarily the same directory as alertmanager itself)

> Let's say I have Team A receiver for Team A resources and Team B receiver for Team B resources, I want to isolate the notifications for each team. Neither team should be aware of or get notified about the other teams resources. How can I accomplish that?

You set labels on the Team A / Team B alerts, and use those labels in routing rules in alertmanager to route them appropriately. But there's no point trying to do that until you've worked out why the system isn't using the configuration file you think it is.

[Chrysale Tchako]

I believe this is the correct config file because when I mute an email from one of the receivers that email is no longer included in any notification that gets sent out. As far as the labels you mentioned earlier, could you share an example of how to implement or format those in my alertmanager.yml and alert.rules.yml files.

[Brian Candler]

Then something strange is going on. Are you 100% sure that you pasted *exactly* the configuration file you're using, with *exactly* the correct contents and indentation?

You can add labels in your service discovery, e.g. if you are using file_sd_configs then inside the targets file you can put

- labels:
    owner: teamA
  targets:
    - foo
    - bar

    - baz

- labels:

    owner: teamB

  targets:

    - qux

In this case, all timeseries which are scraped from the hosts will have those labels added as attributes of every timeseries.

Or you can add labels in your alerting rules:

groups:
- name: UpDown
  rules:
  - alert: UpDownTeamA
    expr: up{instance=~"foo|bar|baz"} == 0
    for: 3m
    keep_firing_for: 3m
    labels:
      owner: teamA
  - alert: UpDownTeamB
    expr: up{instance=~"qux"} == 0
    for: 3m
    keep_firing_for: 3m
    labels:
      owner: teamB

It's up to you on whether you want to label the timeseries themselves (which is simpler but less flexible), or do the labelling in the alerting rules (which means you may need to look at other labels to decide how to route things, but allows you to change your alert routing without changing the underlying timeseries)

In alertmanager you'd then match labels to decide where to deliver.

route:

  # Default receiver (for all alerts that don't match any specific route)

  receiver: "everyone"

  routes:
    - matchers:

        - owner=teamA

      receiver: "gmail"
    - matchers:

        - owner=teamB

      receiver: "dev-receiver"

This simple example says alerts with owner=teamA go to "gmail", alerts with owner=teamB go to "dev-receiver", and everything else goes to "everyone".  (The first match stops any subsequent processing, unless that rule has "continue: true")

There is a tool you can test your alertmanager config with here:

https://prometheus.io/webtools/alerting/routing-tree-editor/

Try pasting in the above block, then click "Draw routing tree". Then next to Match Label Set, enter

{instance="foo",owner="teamA"}

and click Match Label Set.

HTH.

[Chrysale Tchako]

This is the exact configuration of my alermanager.yml file (some values were substituted for privacy/security) do you notice any discrepancies? 

global:
  smtp_hello: 'example.com'

route:
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 12h
  group_by: ["alertname", "severity"]
  # Default receiver (for all alerts that don't match any specific route)

  receiver: "default-receiver"

  routes:
    - receiver: "default-receiver"  # This is for other alerts, not EndpointDown or dev-team alerts
      continue: true
    - match:
        service: "EndpointDown"
      receiver: "custom-receiver"

inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'instance']
    routes:

      - {receiver: default-receiver, continue: true}
      - {receiver: custom-receiver}

receivers:
  # Default receiver for other alerts

  - name: "default-receiver"
    email_configs:
      - to: "exam...@example.com, exam...@example.com"
        from: 'alert-...@example.com'
        smarthost: 'smtp-relay.example.com:587'
        send_resolved: true

  # Receiver for custom alerts (both EndpointDown and dev-team alerts)
  - name: "custom-receiver"
    matchers:
      - service: "EndpointDown"
    email_configs:
      - to: 'us...@example.com'
        from: 'alert-...@example.com'
        smarthost: 'smtp-relay.example.com:587'
        send_resolved: true

[Brian Candler]

There are two major problems with that confgi:

- there is a "routes" block nested under "inhibit_rules", where it is not allowed

- there is a "matchers" block nested under "receivers", where it is not allowed

And if I try to start alertmanager 0.25.0 (the version you say you're using) with that exact configuration, it aborts.

root@prometheus:~# /opt/alertmanager-0.25.0.linux-amd64/alertmanager --config.file=amtest.conf  --web.listen-address=:9193 --cluster.listen-address=0.0.0.0:9194
ts=2025-01-07T08:51:23.716Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)"
ts=2025-01-07T08:51:23.716Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)"
ts=2025-01-07T08:51:23.723Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=10.12.255.33 port=9194
ts=2025-01-07T08:51:23.726Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
ts=2025-01-07T08:51:23.761Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=amtest.conf
ts=2025-01-07T08:51:23.761Z caller=coordinator.go:118 level=error component=configuration msg="Loading configuration file failed" file=amtest.conf err="yaml: unmarshal errors:\n  line 27: field routes not found in type config.plain\n  line 43: field matchers not found in type config.plain"
ts=2025-01-07T08:51:23.761Z caller=cluster.go:690 level=info component=cluster msg="gossip not settled but continuing anyway" polls=0 elapsed=35.105627ms

You cannot possibly be running a working alertmanager with this config.

[Chrysale Tchako]

What modifications would you recommend making to the file? Or better yet, how should it be formatted? 

[Brian Candler]

You should remove the two sections that I highlighted. The following is a valid config, and is accepted:

---- 8< ----

global:
  smtp_hello: 'example.com'


route:
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 12h
  group_by: ["alertname", "severity"]
  # Default receiver (for all alerts that don't match any specific route)
  receiver: "default-receiver"

  routes:
    - receiver: "default-receiver"  # This is for other alerts, not EndpointDown or dev-team alerts
      continue: true
    - match:
        service: "EndpointDown"
      receiver: "custom-receiver"


inhibit_rules:
  - source_matchers:
      - severity="critical"
    target_matchers:
      - severity="warning"
    equal: ['alertname', 'instance']

receivers:
  # Default receiver for other alerts
  - name: "default-receiver"
    email_configs:
      - to: "exam...@example.com, exam...@example.com"
        from: 'alert-...@example.com'
        smarthost: 'smtp-relay.example.com:587'
        send_resolved: true

  # Receiver for custom alerts (both EndpointDown and dev-team alerts)
  - name: "custom-receiver"

    email_configs:
      - to: 'us...@example.com'
        from: 'alert-...@example.com'
        smarthost: 'smtp-relay.example.com:587'
        send_resolved: true

---- 8< ----

But if you tried to use the config you posted, and you think it was accepted, then it must have been reading a different config file.

Note: the config you have written will do the following:

* ALL alerts (including EndpointDown) will be sent to default-receiver

* EndpointDown alerts will additionally be sent to custom-receiver

The receiver under the comment "# Default receiver (for all alerts that don't match any specific route)" is never actually used, since one of the child routes always matches - but you still have to provide it.