Alertmanager to send notifications to google chat
1,335 views
Skip to first unread message
Near N
Jun 12, 2023, 1:59:14 AM6/12/23
to Prometheus Users
Good day. I was wondering if there is a way to configure alertmanager to send notifications to google chat via a webhook.
The current setup that I use consists of vmalert (loaded with set of promQL compatible rules) >> Alertmanager (to group them up) and Calert (to send them to google chat). Despite the flowchart is working fine, I found it difficult to amend proposed message template, so decided to simplify it by configuring alertmanager to send the alerts. It dawns on me that it can operate, but I seem to have failed to fine-tune it. Any help would be appreciated. The alertmanager.yml is below:
global:
http_config:
proxy_url: '{{ alertmanager_proxy_url }}'
route:
receiver: 'my_alerts'
group_wait: 30s
group_interval: 60s
repeat_interval: 15m
group_by: ['room', 'alertName']
receivers:
- name: 'my_alerts'
webhook_configs:
- url: '{{ alertmanager_chat_webhook }}'
send_resolved: true
templates:
- '{{ alertmanager_working_directory }}/google-chat.tmpl'
And here is the error of alertmanager trying to send an alert:
Jun 12 05:26:00 app1-cp alertmanager[140337]: ts=2023-06-12T05:26:00.599Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="my_alerts/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://chat.googleapis.com/v1/spaces/webook_token: {\n \"error\": {\n \"code\": 400,\n \"message\": \"Invalid JSON payload received. Unknown name \\\"receiver\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"status\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"alerts\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"commonLabels\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"commonAnnotations\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"version\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"groupKey\\\" at 'message': Cannot find field.\\nInvalid JSON payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': Cannot find field.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.BadRequest\",\n \"fieldViolations\": [\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"receiver\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"status\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"alerts\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"groupLabels\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"commonLabels\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"commonAnnotations\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"externalURL\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"version\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"groupKey\\\" at 'message': Cannot find field.\"\n },\n {\n \"field\": \"message\",\n \"description\": \"Invalid JSON payload received. Unknown name \\\"truncatedAlerts\\\" at 'message': Cannot find field.\"\n }\n ]\n }\n ]\n }\n}\n"
Brian Candler
Jun 12, 2023, 3:04:17 AM6/12/23
to Prometheus Users
> Invalid JSON payload received.
Alertmanager's webhook sends a JSON message with a *fixed* payload format. See
"The Alertmanager will send HTTP POST requests in the following JSON format to the configured endpoint"
You can't override it with a template. The Alertmanager JSON format is different to what Google Chat requires, and therefore you can't send it directly.
That's why you need a separate piece of software to do the conversion (Calert in your case) - it receives the Alertmanager webhook JSON payload, builds a new payload for the target, and then sends it there.
Near N
Jun 12, 2023, 3:13:50 AM6/12/23
to Prometheus Users
Thanks for reply Brian. I do realize, that Google Chat API requires a particular JSON format, but was assuming Alertmanager is capable of doing it by using particular template, which is wrong. Now the questions is whether it is worth creating a PR to create such a feature or it is out of Alertmanager's standard scope?
Another question is perhaps it is possible to converse the receivable JSON on the hop and just send notifications via curl method? I mean if the endpoint contains the following alert "[{"annotations":{"server":"main","number_of_nodes":"1"},"endsAt":"2023-06-12T07:02:00.233Z","fingerprint":"b21bf244f87cd47f","receivers":[{"name":"my_alerts"}],"startsAt":"2023-06-12T06:44:00.233Z","status":{"inhibitedBy":[],"silencedBy":[],"state":"active"},"updatedAt":"2023-06-12T06:58:00.424Z","generatorURL":"http://app1:8880/vmalert/alert?group_id=71689142540802366","labels":{"alertgroup":"INFRA","alertname":"The number of available Cassandra nodes has changed","server":"main","severity":"Critical"}}]", would it be possible to retrieve this json and post it via curl? This, I assume, wouldn't solve my problem, cause I believe Google API expects something peculiar in terms of the message it receives.
понедельник, 12 июня 2023 г. в 13:04:17 UTC+6, Brian Candler:
Brian Candler
Jun 12, 2023, 6:06:34 AM6/12/23
to Prometheus Users
I can't speak for the project, but my guess would be it's out of scope. There are many existing integrations which use the existing webhook mechanism:
Doing it externally allows the request to be enriched with other data which wouldn't be available in a template, and to use a variety of outbound authentication mechanisms which Alertmanager itself doesn't implement, or even non-HTTP protocols like STOMP.
In any case, text templating is a nasty and insecure way to create JSON or XML payloads.
> it is possible to converse the receivable JSON on the hop and just send notifications via curl method
You'd first have to *receive* the webhook payload from Alertmanager, which involves plugging your curl script into a webserver somehow. You could probably write one as a CGI script under Apache, for example.
But why would you do that, when you have an existing tested and working solution?
Near N
Jun 12, 2023, 6:33:32 AM6/12/23
to Prometheus Users
Brian, thanks for pointing to other integrations. The reason why I was wondering about curling the messages is that the existing solution (Calert) provides little information about templating options. Alerts are incoming, but with tight information. I'd love to highlight severity in different colors, add warning icons, e.g. something similar to Zabbix or at least that can suit basic user-friendly message.
понедельник, 12 июня 2023 г. в 16:06:34 UTC+6, Brian Candler:
Brian Candler
Jun 12, 2023, 9:43:10 AM6/12/23
to Prometheus Users
If we're talking about the same "calert" then it looks like it already supports user-supplied templating of the message content:
The sample config shows how to configure it:
If you want to do colours and highlighting, it would depend whether the Google Chat API accepts HTML tags, or some other way of marking up the content. You'll need to find that out.
The variables available in the templates look to be very similar to those provided in Alertmanager templates:
In fact, there's a test case which uses exactly the alertmanager code (an instance of alertmgrtmpl.Alert) to drive it:
Near N
Jun 14, 2023, 12:12:53 AM6/14/23
to Prometheus Users
Correct, that's same calert I was mentioning previously. I understand the use case of default template, but thought it is possible to do highlighting within the template. It is basically possible by converting the message in a wildcard structure acceptable by Google Chat API, but I couldn't find any options to do so with calert template, but thanks for suggestions
понедельник, 12 июня 2023 г. в 19:43:10 UTC+6, Brian Candler:
Brian Candler
Jun 14, 2023, 2:21:49 AM6/14/23
to Prometheus Users
> It is basically possible by converting the message in a wildcard structure acceptable by Google Chat API
I don't know what you mean by a "wildcard structure". Can you give an example, and/or point to documentation? Is it using tags within the message field, or does it require a different JSON structure entirely?
Near N
Jun 14, 2023, 3:14:03 AM6/14/23
to Prometheus Users
Brian, a wildcard is just a gibberish of standard card messages proposed by Google. I believe you're already familiar with https://developers.google.com/chat/api/guides/message-formats/cards#apps-script
Based on Mr-Karan's example (https://github.com/mr-karan/calert/blob/main/docs/examples/send_alert.sh) my assumption is as follows:
1) Calert retrieves an unstructured json alert, which alertmanager puts to host:6000
2) After that calert converts such a raw json to the one suiting it's template
3) It is then posting it to a google chat via curl
Hence, calert is capable of processing the message under the hood, converting it to Google API-friendly json format and this is I guess I was looking for, but lack general understanding of whether it is possible to amend
среда, 14 июня 2023 г. в 12:21:49 UTC+6, Brian Candler:
Brian Candler
Jun 14, 2023, 5:00:43 AM6/14/23
to Prometheus Users
That's clearer, so there are two API formats: a "text" message and a "card" message.
It looks like Calert is hard-coded to using the "text" format:
Hence you could either extend Calert to work with cards too - or raise it as a feature request; or you could implement your own alertmanager webhook to Google Chat gateway.
Reply all
Reply to author
Forward
0 new messages