Alert Manager routes

[Pete Leese]

Hi All,

Trying to make some changes to my alertmanager.yml 

- Send all Critical & Warning alerts to OpsGenie

- Send all Critical & Warning alerts to Slack

- Send all information alerts to email

- DeadMansSwitch - OpsGenie Heartbeat 

Getting in a bit of muddle with it to be honest :) and cannot get the amtool to validate the routes :) 

Would anyone be able to eye-ball the below and advise? Many thanks :) 

global:

  resolve_timeout: 5s

  smtp_smarthost: 'xxxx25'

  smtp_from: 'xxxxx'

  smtp_require_tls: false

route:

  receiver: slack_cloud_alerts

  group_by: [alertname, cluster]

  group_wait: 5s

  group_interval: 10s

  repeat_interval: 3h

  routes:

  - match:

      severity: critical 

    receiver: slack_cloud_alerts

    continue: true

  - match:

      severity: critical

    receiver: opsgenie_cloud_alerts

    continue: true

  - match:

      severity: warning

    receiver: opsgenie_cloud_alerts

    continue: true

  - match:

      severity: information

    receiver: email_cloud_alerts

    continue: true

   receiver: deadmansswitch

  - match:

      alertname: DeadMansSwitch

    repeat_interval: 1m

receivers:

- name: email_cloud_alerts

  email_configs:

  - to: xxxxxxx

    require_tls: false

- name: slack_cloud_alerts

  slack_configs:

  - api_url: https://hooks.slack.com/services/xxxxxxxxxxxxxx

    send_resolved: true

    channel: 'xxxxxx'

    username: 'xxxxx'

    icon_url: 'https://ca.slack-edge.com/xxxxxxxxxxxxxxxx'

    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

    pretext: '{{ .CommonAnnotations.summary }}'

    text: |-

      {{ range .Alerts }}

         *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`

        *Description:* {{ .Annotations.description }}

        *Details:*

        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`

        {{ end }}

        *Grafana:* <https://xxxxxxxxxxxxxxxx?refresh=30s&orgId=1&var-interval=5m&var-server={{ reReplaceAll "(.*):(.*)" "${1}" .Labels.instance }} |:chart_with_upwards_trend:>

        *Runbook:* <https://xxxxxxxxxxxxxxx/{{ reReplaceAll "(.*):(.*)" "${1}" .Labels.alertname }}.aspx  |:spiral_note_pad:>

        {{ end }}

    footer: "Prometheus Alerts"

- name: slack_eol_alerts

  slack_configs:

  - api_url: https://hooks.slack.com/services/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

    send_resolved: true

    channel: '#eol-alerts'

    username: 'Cobot'

    icon_url: 'https://ca.slack-edge.com/T025JBPLS-U9JEG8NNP-69faa399bab4-72'

    title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"

    text: "*Details:* {{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"

    footer: "Prometheus Alerts"

- name: opsgenie_cloud_alerts

  opsgenie_configs:

  - api_url: https://api.eu.opsgenie.com  

    api_key: xxxxxxxxxxxxxxxxx

    teams: xxxxxxxxxxx   # Put in your team here

- name: deadmansswitch

  webhook_configs:

  - url: 'https://api.eu.opsgenie.com/v2/heartbeats/PrometheusAlertManager/ping'

    send_resolved: true

    http_config:

      basic_auth:

        password: xxxxxxxxxxxxxxxx

[Simon Pasquier]

You can configure OpsGenie & Slack in the same receiver section which
would simplify your configuration IMO.
BTW you'll need to reset your slack webhook URL...

- name: slack_opsgenie

slack_configs:
- api_url: https://hooks.slack.com/services/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
send_resolved: true
channel: '#eol-alerts'
username: 'Cobot'
icon_url: 'https://ca.slack-edge.com/T025JBPLS-U9JEG8NNP-69faa399bab4-72'
title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
text: "*Details:* {{ range .Alerts }}{{ .Annotations.description
}}\n{{ end }}"
footer: "Prometheus Alerts"

opsgenie_configs:
- api_url: https://api.eu.opsgenie.com
api_key: xxxxxxxxxxxxxxxxx
teams: xxxxxxxxxxx # Put in your team here

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3a29daa7-6786-47d2-b388-abc875c3fed8%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Pete Leese]

How would I go about doing that? 

Can you see any issues as to why this configuration would not pass the amtool checks? 

Gah - I missed that - will reset now :) 

P

[Pete Leese]

Worth noting that I only send critical alerts to slack currently.

[Pete Leese]

Hi all,

Is anyone able to point me in the right direction please ? :)

[Simon Pasquier]

Hi Pete,
Not sure what's your question actually. Does the configuration load
properly now?

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/47b4d1c1-f5b2-4fb8-ac29-8c08d0fdec65%40googlegroups.com.

[Pete Leese]

Hi Simon,

The configuration loads - however fails the amtool checks - so I really wasn't sure if it was working correctly as was seeing some issues with alerts being received consistently hence reaching out for someone to check the config I have was sane. 

Incidentally - would you know how to disable group_by for alerts sent to OpsGenie - Currently they are being received group together which is not ideal. 

Cheers

Pete 

[Simon Pasquier]

On Thu, Oct 18, 2018 at 1:48 PM 'Pete Leese' via Prometheus Users
<promethe...@googlegroups.com> wrote:
>
> Hi Simon,
>
> The configuration loads - however fails the amtool checks - so I really wasn't sure if it was working correctly as was seeing some issues with alerts being received consistently hence reaching out for someone to check the config I have was sane.

That would help if you pasted the errors you get from amtool.

>
> Incidentally - would you know how to disable group_by for alerts sent to OpsGenie - Currently they are being received group together which is not ideal.

This isn't possible right now. If you want to "ungroup" alerts then
you will have to pass all the potential label names to the "group_by:"
clause.

>
> Cheers
>
> Pete

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b2b699d9-3d76-4f1e-981a-431e09ce5ddc%40googlegroups.com.

[Pete Leese]

Ok - Maybe I'll have to disable group_by for now and raise a suggestion on GitHub...

afraid the amtool doesn't give much information.

amtool: error: unexpected routes.

[Simon Pasquier]

On Thu, Oct 18, 2018 at 5:20 PM 'Pete Leese' via Prometheus Users
<promethe...@googlegroups.com> wrote:
>
> Ok - Maybe I'll have to disable group_by for now and raise a suggestion on GitHub...

There's already https://github.com/prometheus/alertmanager/pull/1588
But frankly, I don't think there's enough agreement from the
maintainer team to get this in.

>
> afraid the amtool doesn't give much information.
>
> amtool: error: unexpected routes.

Well it tells you that a "routes:" key isn't at the correct level.
Please pastebin the configuration file and the amtool version.

>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/85edb420-edba-41ca-8685-fd5ff795854d%40googlegroups.com.