Prometheus/AlertManager to Discord

[Adam]

Hi all,

I am using alertmanager notifications with slack and email and everything is ok. Now I am trying to send the same notifications to a Discord server too via Webhooks.

Using the below configurations in "/etc/alertmanager/alertmanager.yml", I can receive alerts on Slack and Email but not Discord. Please note that I tested the Discord webhook URL in other tools and it works fine.

"global:

  smtp_smarthost: 'smtp.gmail.com:587'

  smtp_from: 'AlertManager <x...@gmail.com>'

  smtp_require_tls: true

  smtp_hello: 'alertmanager'

  smtp_auth_username: 'xyz'

  smtp_auth_password: 'XXXXXXXXXXXXXXX'

  slack_api_url: 'https://hooks.slack.com/services/XYZ/ABC/EFG5'

route:

  group_by: ['instance', 'alert']

  group_wait: 30s

  group_interval: 5m

  repeat_interval: 1h

  receiver: 'R1'

receivers:

  - name: 'R1'

#    email_configs:

#      - to: 'a...@gmail.com'

    slack_configs:

      # https://prometheus.io/docs/alerting/configuration/#slack_config

      - channel: 'system_events'

      - username: 'AlertManager'

    webhook_configs:

      - url: 'https://discordapp.com/api/webhooks/XYZ/ABC'

My questions is:

1-Why I am not receiving alerts Discord?
2-Is there a better way to achieve the same thing(Discord Integration)?

Thank you in advance.

[Brian Candler]

On Sunday, 13 September 2020 14:59:39 UTC+1, Adam wrote:

1-Why I am not receiving alerts Discord?

You'll need to check logs to answer that.  Not knowing anything about discord, I'd say the most likely problems are:

1. discord's API does not accept the JSON object structure which prometheus sends in its webhook POST (if so, I'd expect discord to send a 400 response).  The format that Prometheus sends is fixed, so if Discord needs a different format, you'd need to write some sort of proxy which translates it.

 

2. discord's API requires some sort of authentication, e.g. a bearer token, which you've not configured (if so, I'd expect discord to send a 401 or 403 response)

The response code should be visible in alertmanager logs.

 

2-Is there a better way to achieve the same thing(Discord Integration)?

Discord appears to be a commercial service so probably best to ask their support, or on their discussion group if there is one.

[Adam]

Thank you for pointing me to the right direction. I checked logs and found the following error message. 

cancelling notify retry for \"webhook\" due to unrecoverable error: unexpected status code 400: https://discordapp.com/api/webhooks/YYYYYYY/XXXXXXX"

I googled this error and found this thread https://stackoverflow.com/questions/55792653/adding-custom-webhook-configuration-in-alertmanager but I am not sure how can I change HTTP METHOD to POST.

Thank you

[Brian Candler]

You've misunderstood that stackoverflow response.  The poster had written their own webhook receiver, which was expecting GET, but they needed to change it to receive POST.

In your case, you want to use someone else's webhook receiver (discord).  That already expects POST.  However, the Alertmanager webhook sends JSON in a fixed payload format.  The Discord webhook receiver requires JSON in a different, fixed payload format.  They are not compatible with each other.

There seems to be a misunderstanding that if any two things have an API, those two things will automatically be able to talk to each other.  That's simply not true.  These are two different APIs, even though they're both described as a "webhook".

You will need to write some software which receives an Alertmanager webhook POST, reformats it, and re-sends it as a Discord webhook POST.