Multiple destinations receiver

[Martin]

Hi there, this is not working for me. 

Every notification is sent to either one of the destinations. 

I keep receiving or a mail or a telegram message for every firing alert.

Seems to be a kind of round robin. It makes no sense for me.

Any ideas? Have to manage it necessarily at route level?

Thank you all,

receivers:
  - name: "devops"
    email_configs:
      - to: "ma...@mail.com"
        send_resolved: false
        html: '{{ template "mail.template.html" . }}'
        headers:
          subject: '{{ template "mail.template.subject" . }}'
    telegram_configs:
      - send_resolved: false
        api_url: https://api.telegram.org
        bot_token: token.....
        chat_id: -123235456.....
        message: '{{ template "telegram.template.message" . }}'
        disable_notifications: false
        parse_mode: ""

[Brian Candler]

What alertmanager version?  Do you have any sort of alertmanager clustering, and if so are you sure the nodes are consistent?

As far as I know, what you've written is file.  Rules which send to receiver "devops" *should* send to both destinations.

Therefore if this isn't happening then you should look at alertmanager logs, to see what it says is happening at delivery time.  You should also look at the metrics that alertmanager itself generates; you can scrape these with a separate scrape job, e.g.

  - job_name: alertmanager
    scrape_interval: 1m
    metrics_path: /metrics
    static_configs:
      - targets: ['localhost:9093']

This will give you counters for delivery attempts and failures for each delivery method, e.g.

alertmanager_notification_latency_seconds_bucket{integration="telegram",le="1"} 0
alertmanager_notification_latency_seconds_bucket{integration="telegram",le="5"} 0
alertmanager_notification_latency_seconds_bucket{integration="telegram",le="10"} 0
alertmanager_notification_latency_seconds_bucket{integration="telegram",le="15"} 0
alertmanager_notification_latency_seconds_bucket{integration="telegram",le="20"} 0
alertmanager_notification_latency_seconds_bucket{integration="telegram",le="+Inf"} 0
alertmanager_notification_latency_seconds_sum{integration="telegram"} 0
alertmanager_notification_latency_seconds_count{integration="telegram"} 0
alertmanager_notification_requests_failed_total{integration="telegram"} 0
alertmanager_notification_requests_total{integration="telegram"} 0
alertmanager_notifications_failed_total{integration="telegram"} 0
alertmanager_notifications_total{integration="telegram"} 0

Aside: if you had two different receivers, say "devops_email" and "devops_telegram", then it's possible to write a routing rule which sends to both:

    # OLD

    - matchers:

        - 'foo = bar'

      receiver: devops

    # NEW

    - matchers:

        - 'foo = bar'

      routes: [ {receiver: devops_mail, continue: true}, {receiver: devops_telegram} ]

However what you wrote originally should work fine as far as I know.

[Brian Candler]

> As far as I know, what you've written is file

s/file/fine/

[Martin]

Despite the conf was correct it never worked fine.

With matchers instead of match_re it works like a charm :)

Thanks ;)

[Brian Candler]

On Thursday, 23 June 2022 at 13:37:02 UTC+1 Martin wrote:

Despite the conf was correct it never worked fine.

With matchers instead of match_re it works like a charm :)

That's weird.  I never had a problem with match_re.  I just moved to matchers because it's the "new" way of doing things, and more configurable.

You haven't show your routing rules, but if you can find a way to reproduce this, with a current version of alertmanager, then it should be raised as a bug.

You are sure it's not just a YAML formatting or alignment problem?  However, I can't see how it could cause alerts to be delivered to one destination but not another intermittently, when both destinations are under the same "receiver".