alertmanager send sms

427 views
Skip to first unread message

bgz

unread,
Dec 3, 2020, 5:42:35 PM12/3/20
to Prometheus Users
Does anyone have an actual working example config for sending out sms. I have a custom webhook but i can't workout where do i actually put the mobile numbers in the alertmanger.yml file. I have checked official documentation and searched git for a working example but whatever i try i end up with error:

file=/etc/alertmanager/alertmanager.yml err="yaml: unmarshal errors:\n  line 51: field to not found in type config.plain\n  line 53: field send_resolved not found in type config.plain"

My config:

receivers:

- name: 'opsteam-sms'

   - to: '+2111221121'

      send_resolved: true

  webhook_configs:

   - url: 'http://loclahost:9091/send'


b.ca...@pobox.com

unread,
Dec 4, 2020, 3:35:54 AM12/4/20
to Prometheus Users
What software is receiving your webhook request and sending the SMS?  In other words, what software is listening on port 9091?  Most likely you need to put your SMS number in that configuration somewhere, or maybe as part of the URL.  And of course, it must be happy to receive requests in the prometheus webhook JSON format.

Notice that your error message says: "field to not found in type..."  In other words, you put a "to:" setting where it was not expected.

The settings you are allowed to put under "receivers" are documented here: https://prometheus.io/docs/alerting/latest/configuration/#receiver
The settings you are allowed to put under "webhook_configs" are documented here: https://prometheus.io/docs/alerting/latest/configuration/#webhook_config
Neither of them allows "to".  Note also that send_resolved belongs under webhook_configs, not receivers, and that you mistyped "localhost".

Fixing these gives:

receivers:
- name: 'opsteam-sms'
  webhook_configs:
   - url: 'http://localhost:9091/send?recipient=+2111221121'   # a complete guess - it depends on your SMS gateway software
     send_resolved: true

bgz

unread,
Dec 4, 2020, 3:53:21 AM12/4/20
to Prometheus Users
 Thanks for the reply i am going to try this. But how does this scale? Let's say i have a smaller set or or rules and i want to have for one of them 10 numbers where i send then for the second i want 5 numbers and so on? Do i look i look into having 20 url's with numbers in the receivers section like your example? The documentation is not clear to me at all i spent days trying going thorough it. 

b.ca...@pobox.com

unread,
Dec 4, 2020, 6:30:17 AM12/4/20
to Prometheus Users
You haven't said what SMS software you're using, nor linked to its documentation, so I have no idea how it would scale.

Maybe it lets you list multiple SMS numbers in the URL?  Maybe you have to invoke the webhook multiple times with different numbers?  Note that you can invoke multiple webhooks for a single alertmanager receiver, because it's a list of webhook_configs. e.g.

receivers:
- name: 'opsteam-sms'
  webhook_configs:
     send_resolved: true
     send_resolved: true

That will trigger two webhooks for any alert routed to "opsteam-sms"

b.ca...@pobox.com

unread,
Dec 4, 2020, 6:33:19 AM12/4/20
to Prometheus Users
On Friday, 4 December 2020 at 08:53:21 UTC bgz wrote:
 Let's say i have a smaller set or or rules and i want to have for one of them 10 numbers where i send then for the second i want 5 numbers and so on?

Then you have two receivers, one which delivers to 10 numbers and one which delivers to 5 numbers, and you use the routing tree to choose the right receiver for a given alert.  You might do this by matching the alertname explicitly, or you might do this by having labels set on certain alert rules.
Reply all
Reply to author
Forward
0 new messages