discord - "unrecoverable error"

[Emil Fabrice Semti]

Hi Everyone,

Seeking some help here - very new to Prometheus/Alert Manager, must add.

- I set up the kube-prometheus-stack as per: https://github.com/fabricesemti80/home-stack/blob/42f23c6e62184b037908e84d52776a717889ff20/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml#L81

- log errors are similar to this:

ts=2022-08-30T06:50:59.456Z caller=dispatch.go:354 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="discord/slack[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"#prometheus\": unexpected status code 400: {\"attachments\": [\"0\"]}"

- I used this site to test the webhook

- the test - with the same parameters used in the manifest - was successful (see attached screenshot), therefore I think my settings are correct.

Any suggestions what am I missing please?

Thanks,
F.

[Brian Candler]

You haven't said what you've configured as the receiver (target) of the webhook.

Whatever it is, it is responding with a 400 error, which means that the request being sent is invalid.

The format of the webhook payload sent by Alertmanager is *fixed*.  For an example see:

https://prometheus.io/docs/alerting/latest/configuration/#webhook_config

This will be different to what a "Discord webhook tester" sends.

Therefore, if you're pointing the Prometheus alertmanager webhook receiver *directly* at a Discord API, it definitely won't work. You will need some middleware to translate the Alertmanager webhook payload into something the target API understands.  Google "alertmanager discord" for some options.

As for sending to slack, you should use the Alertmanager built-in slack integration, not webhook:

https://prometheus.io/docs/alerting/latest/configuration/#slack_config

You can of course make one alertmanager receiver which does both, which is perhaps what you're already doing:

receivers:

- name: discord/slack
  slack_configs:
    - ...

  webhook_configs:

    - ...

[Fabrice Semti]

Best regards,

Fabrice Semti

📧 fab...@fabricesemti.com   📧 emilf...@gmail.com 
🌐 fabricesemti.com

---------- Forwarded message ---------
Feladó: Fabrice Semti <fab...@fabricesemti.com>
Date: 2022. aug. 30., K, 13:42
Subject: Re: [prometheus-users] Re: discord - "unrecoverable error"
To: Brian Candler <b.ca...@pobox.com>

Hi Brian,

Thanks for the reply.

You haven't said what you've configured as the receiver (target) of the webhook.

So I have created a webhook on my discord channel. (The same webhook was tested on the online tester)

Therefore, if you're pointing the Prometheus alertmanager webhook receiver *directly* at a Discord API, it definitely won't work. You will need some middleware to translate the Alertmanager webhook payload into something the target API understands.  Google "alertmanager discord" for some options.

I have based my config on this one, which seems to be in line with what I found for "alertmanager discord"...

My config is:

  alertmanager:

      config:

        global:

          slack_api_url: "${SECRET_ALERT_MANAGER_DISCORD_WEBHOOK}"

          resolve_timeout: 5m

        receivers:

          - name: "null"

          - name: "discord"

            slack_configs:

              - channel: "#prometheus"

                icon_url: https://avatars3.githubusercontent.com/u/3380462

                username: "Prometheus"

                send_resolved: true

                title: |-

                  [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ if ne .CommonAnnotations.summary ""}}{{ .CommonAnnotations.summary }}{{ else }}{{ .CommonLabels.alertname }}{{ end }}

                text: >-

                  {{ range .Alerts -}}

                    **Alert:** {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

                  **Description:** {{ if ne .Annotations.description ""}}{{ .Annotations.description }}{{else}}N/A{{ end }}

                  **Details:**

                    {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`

                    {{ end }}

                  {{ end }}

        route:

          group_by: ["alertname", "job"]

          group_wait: 30s

          group_interval: 5m

          repeat_interval: 6h

          receiver: "discord"

          routes:

            - receiver: "null"

              match:

                alertname: Watchdog

            - receiver: "discord"

              match_re:

                severity: critical|warning|error

              continue: true

        inhibit_rules:

          - source_match:

              severity: "critical"

            target_match:

              severity: "warning"

            equal: ["alertname", "namespace"]

      ingress:

        enabled: true

        pathType: Prefix

        ingressClassName: "nginx"

        annotations:

          cert-manager.io/cluster-issuer: "letsencrypt-production"

          external-dns.alpha.kubernetes.io/target: "ipv4.${SECRET_DOMAIN}"

          external-dns/is-public: "true"

          hajimari.io/enable: "true"

          hajimari.io/icon: "fire-alert"

          # nginx.ingress.kubernetes.io/auth-url: "https://oauth.${SECRET_DOMAIN}/oauth2/auth"

          # nginx.ingress.kubernetes.io/auth-signin: "https://oauth.${SECRET_DOMAIN}/oauth2/start?rd=$scheme://$best_http_host$request_uri"

          # nginx.ingress.kubernetes.io/auth-response-headers: "x-auth-request-user, x-auth-request-email, x-auth-request-access-token"

        hosts:

          - &host "alert-manager.${SECRET_DOMAIN}"

        tls:

          - hosts:

              - *host

            secretName: "alert-manager-tls"

      alertmanagerSpec:

        storage:

          volumeClaimTemplate:

            spec:

              storageClassName: "hdd-nfs-storage"

              resources:

                requests:

                  storage: 1Gi

      resources:

        requests:

          cpu: 11m

          memory: 37M

You can of course make one alertmanager receiver which does both, which is perhaps what you're already doing:

receivers:

- name: discord/slack
  slack_configs:
    - ...

  webhook_configs:

    - ...

Yes, pretty much that is what I am aiming for, and I only lightly modified what I have seen in other repos (such as above or this one ), so I am not sure what is not right...

Any further clues would be much appreciated...

 

Best regards,

Fabrice Semti

📧 fab...@fabricesemti.com   📧 emilf...@gmail.com 
🌐 fabricesemti.com

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/uqNf-alj0i0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e7f7f14f-514f-474c-8052-5bae2c0a6a64n%40googlegroups.com.

[Brian Candler]

I think the part you've missed is that an *alertmanager* webhook message is completely different to a *discord* webhook message.  You need an application in between to translate - that is, it receives an alertmanager webhook message, and send it on to discord with a different payload.

If you google "alertmanager discord" then the first two hits are projects which do exactly this:

https://github.com/benjojo/alertmanager-discord

https://github.com/rogerrum/alertmanager-discord

Now, you linked to this bit of helm chart:

https://github.com/cbirkenbeul/k3s-gitops/blob/main/cluster/apps/monitoring/kube-prometheus-stack/helmrelease.yaml

That doesn't use alertmanager webhooks at all.  It uses the *slack* sending capability of alertmanager.

Does Discord accept Slack-formatted API messages?  If so, then it might work.  This suggests that it could:

https://stackoverflow.com/questions/66076875/discord-slack-compatible-webhook

But there's a separate API endpoint for accepting Slack-format requests:

https://discord.com/developers/docs/resources/webhook#execute-slackcompatible-webhook

So either:

1. you're not POSTing to the correct endpoint (the Slack-compatible one)

2. something about alertmanager's request makes Discord unhappy (i.e. it's not as Slack-compatible as it claims to be)

If you think it's case (2) then you can take it up with Discord support - although they probably will want to see the body of the HTTP request, and the body of the 400 response that comes back.

[Fabrice Semti]

Thanks for your reply.

I mean, if I replace the SECRET_ALERT_MANAGER_DISCORD_WEBHOOK with SECRET_ALERT_MANAGER_SLACK_WEBHOOK(so using instead of a Discord webhook an actual Slack one - to a freshly created slack channel of mine)  the alerts are indeed coming in..

So indeed something must be amiss between alert manager and/or Discord...will do some reading / reaching out to Discord support, I thought the Discord webhook by default interchangeable with the Slack-one, at least the helmreleases I have seen seemed to indicate this.

Will circle back here if that avenue is exhausted...

Best regards,

Fabrice Semti

📧 fab...@fabricesemti.com   📧 emilf...@gmail.com 
🌐 fabricesemti.com

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/33a1a588-f170-4b26-9285-fdc7db3fd509n%40googlegroups.com.

[Brian Candler]

And the URL you've configured, it definitely looks like

    /webhooks/{webhook.id}/{webhook.token}/slack

?

[Fabrice Semti]

Sorry for the delay. Updated my config based on your suggestion, and indeed I think that fixed it so I am very grateful, thank you very much!

          slack_api_url: "${SECRET_ALERT_MANAGER_DISCORD_WEBHOOK}/slack" ## ? Discord

          # slack_api_url: "${ SECRET_ALERT_MANAGER_SLACK_WEBHOOK}" ## ? Slack

Was just waiting for Prom to fire an alert to Discord after the changes, which just happened.

Thanks again for your help!

Best regards,

Fabrice Semti

📧 fab...@fabricesemti.com   📧 emilf...@gmail.com 
🌐 fabricesemti.com

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e4768237-b7b0-47d6-96c2-a0bd4e454ed1n%40googlegroups.com.