Basic webhook for alertmanager/prometheus

[Danny de Waard]

As a complete noob i want to know how i can simply introdcue a webhook config for prometheus/alertmanager.
For certain alerts i want to send a webhook with 3 parameters to a server.
How do i implement this in prometheus/alertmanager.

I allready have some recievers in place....

receivers:

  - name: 'default'

  - name: 'mail'

    email_configs:

      - send_resolved: true

        to: maila...@domain.nl ##fill in your email

        require_tls: false

  - name: 'all'

    email_configs:

      - send_resolved: true

        to: some...@domain.nl ##fill in your email

        require_tls: false

      - send_resolved: true

        to: "01234...@sms.domain.nl" ##fill in your phonenumbers

        require_tls: false

  - name: 'webhook'

    webhook_configs:

      - url: 'http://127.0.0.1:9000'

[Brian Candler]

The format of the webhook JSON body is fixed.  You can write a small proxy which receives this JSON as input, and sends whatever you like to the downstream server.

[Danny de Waard]

I have a webhook service in place that works. And i can send variables to a script. But how do i get AlertManager (prometheus) to send something to that wqwebhook based on a rule.

Op vrijdag 27 maart 2020 08:32:06 UTC+1 schreef Brian Candler:

[Brian Candler]

To control which receivers get which alerts, you need to configure "routes".  You haven't shown that part of your configuration.

[Danny de Waard]

route:

    # A default reciever
    receiver: 'default'

    # The child route trees.
    routes:
    # This routes performs a regular expression match on alert labels to
    # catch alerts that are related to a list of services.
    - match:
        severity: warning
      receiver: 'mail'
    - match:
        severity: critical
      receiver: 'all'


Op vrijdag 27 maart 2020 12:36:56 UTC+1 schreef Brian Candler:

[Brian Candler]

Therefore you need to include your webhook receiver in some routes too.  Here's one option:

  routes:

    # Send everything to webhook

    - receiver: 'webhook'

      continue: true

    - match:
        severity: warning
      receiver: 'mail'
    - match:
        severity: critical
      receiver: 'all'

Or maybe you only want webhook to be used for alerts which *don't* match the E-mail, in which case you can move "receiver: webhook" to the end.

Another option is to put your webhook_config under other receivers as appropriate:

receivers:

  - name: 'default'

    webhook_configs:

      - url: 'http://127.0.0.1:9000'

  - name: 'mail'

    email_configs:

      - send_resolved: true

        to: maila...@domain.nl ##fill in your email

        require_tls: false

    webhook_configs:

      - url: 'http://127.0.0.1:9000'

  - name: 'all'

    email_configs:

      - send_resolved: true

        to: some...@domain.nl ##fill in your email

        require_tls: false

      - send_resolved: true

        to: "01234...@sms.domain.nl" ##fill in your phonenumbers

        require_tls: false