SMS alert message to Mobile phone

mohd

unread,

Nov 19, 2019, 11:26:34 AM11/19/19

to Prometheus Users

Hello Team,

We have setup prometheus in our environment with Node_exporter. We use alert manager to send email message to our team. We are receivin email alerts if any services are down or threshold exceeds etc .

How to send alert as SMS?

Here is my alertmanager config.yml

route:

repeat_interval: 1d

group_interval: 1d

group_by: [Alertname]

# Send all notifications to me.

receiver: email-me

receivers:

- name: email-me

email_configs:

- to: rece...@gmail.com

from: sen...@gmail.com

smarthost: smtp.gmail.com:587

auth_username: "sen...@gmail.com"

auth_identity: "sen...@gmail.com"

auth_password: "password"

I googled out, there need to be set webhook_config

https://prometheus.io/docs/alerting/configuration/#webhook_config

I am not aware where I need to set this webhook_config, under alertmanager conf or any other config file

For slack I have created incoming webhook url and place that in grafana and I am able to receive slack notification for any alerts.

Kindly share some light on the above subject.

Thank you.

Simon Pasquier

unread,

Nov 20, 2019, 3:16:32 AM11/20/19

to mohd, Prometheus Users

On Tue, Nov 19, 2019 at 5:26 PM mohd <mohdzak...@gmail.com> wrote:
>
> Hello Team,
>
> We have setup prometheus in our environment with Node_exporter. We use alert manager to send email message to our team. We are receivin email alerts if any services are down or threshold exceeds etc .
>
> How to send alert as SMS?

Look here:
https://prometheus.io/docs/operating/integrations/#alertmanager-webhook-receiver

There's one entry for SMS:
https://github.com/messagebird/sachet

>
> Here is my alertmanager config.yml
> route:
> repeat_interval: 1d
> group_interval: 1d
> group_by: [Alertname]
> # Send all notifications to me.
> receiver: email-me
> receivers:
> - name: email-me
> email_configs:
> - to: rece...@gmail.com
> from: sen...@gmail.com
> smarthost: smtp.gmail.com:587
> auth_username: "sen...@gmail.com"
> auth_identity: "sen...@gmail.com"
> auth_password: "password"
>
> I googled out, there need to be set webhook_config
> https://prometheus.io/docs/alerting/configuration/#webhook_config
> I am not aware where I need to set this webhook_config, under alertmanager conf or any other config file

Because Alertmanager doesn't support SMS notifications natively, you
need to run an additional service that will receive JSON notifications
from Alertmanager and forward them to the SMS sender. Usually this
service will run next to Alertmanager.

The webhook receiver will be listed under "receivers", just like any
other email or slack receiver.

route:
repeat_interval: 1d
group_interval: 1d
group_by: [Alertname]

receiver: webhook
receivers:
- name: webhook
webhook_configs:
- url: http://localhost:8080/

If you want notifications to go both to email and SMS, you can have a
multi-destination receiver:

receivers:
- name: sms-email
webhook_configs:
- url: http://localhost:8080/
email_configs:
- ...

>
>
> For slack I have created incoming webhook url and place that in grafana and I am able to receive slack notification for any alerts.
>
>
> Kindly share some light on the above subject.
>
> Thank you.
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3e21fb50-8c46-4c63-8405-4b8141ad677b%40googlegroups.com.

mohd

unread,

Nov 26, 2019, 9:18:03 AM11/26/19

to Prometheus Users

Thanks Simon,

I tried https://github.com/messagebird/sachet

Alertmanager.conf

route:

repeat_interval: 1d

group_interval: 1d

group_by: [Alertname]

# Send all notifications to me.

receiver: email-me

receivers:

- name: email-me

email_configs:

- to: rece...@gmail.com

from: sen...@gmail.com

smarthost: smtp.gmail.com:587

auth_username: "sen...@gmail.com"

auth_identity: "sen...@gmail.com"

auth_password: "password"

- name: 'team-sms'

webhook_configs:

- url: 'http://10.10.10.xx:9876/alert'

to:

- '+919123456789'

from: '+919876543211'

And also tried to do with aws sns, created sns topic and iam role

url: http://<forwarder_url>/alert/<sns_topic_name> - Not sure about forwarder_url

#- name: 'sns-forwarder'

# webhook_config:

#- send_resolved: True

# url: http://alertmanager-sns-forwarder-svc.default:9087/alert/SMSTopic

docker-compose.yml

version: "3.2"

services:

my-monitoring:

image: /my-monitoring

user: "root"

depends_on:

- prometheus

- influxdb

ports:

- "3000:3000"

networks:

- back-tier

- front-tier

restart: "always"

links:

- influxdb:influxdb

- prometheus:prometheus

volumes:

- type: volume

source: ./grafana/data

target: /var/lib/grafana

- type: volume

source: ./grafana/provisioning

target: /etc/grafana/provisioning/

env_file:

- ./grafana/conf/config.monitoring

influxdb:

image: influxdb

user: "root"

ports:

- "8086:8086"

networks:

- back-tier

restart: "always"

volumes:

- type: volume

source: ./influx/data

target: /var/lib/influxdb

prometheus:

image: prom/prometheus:latest

user: "root"

volumes:

- ./prometheus/:/etc/prometheus/

- ./prometheus/data:/prometheus

depends_on:

- influxdb

command:

- '--config.file=/etc/prometheus/prometheus.yml'

- '--storage.tsdb.path=/prometheus'

- '--web.console.libraries=/usr/share/prometheus/console_libraries'

- '--web.console.templates=/usr/share/prometheus/consoles'

ports:

- 9090:9090

links:

- influxdb:influxdb

- node-exporter:node-exporter

- cadvisor:cadvisor

# - blackbox_exporter

networks:

- back-tier

restart: always

telegraf:

container_name: telegraf

image: telegraf:1.9.4-alpine

networks:

- back-tier

volumes:

- ./telegraf/telegraf.conf:/etc/telegraf/telegraf.conf

- /var/run/docker.sock:/var/run/docker.sock

links:

- influxdb:influxdb

restart: always

node-exporter:

image: prom/node-exporter

volumes:

- /proc:/host/proc:ro

- /sys:/host/sys:ro

- /:/rootfs:ro

command:

- '--path.procfs=/host/proc'

- '--path.sysfs=/host/sys'

- --collector.filesystem.ignored-mount-points

ports:

- 9100:9100

networks:

- back-tier

restart: always

deploy:

mode: global

cap_add:

- ALL

black-exporter:

image: prom/blackbox-exporter

hostname: black-exporter

volumes:

- ./config/blackbox.yml:/opt/data/monitoring/docker-composer-ubuntu-v2/blackbox/blackbox.yml

- ./blackbox/blackbox.yml:/config/blackbox.yml

command:

- '--config.file=/config/blackbox.yml'

ports:

- '9115:9115'

networks:

- back-tier

links:

- prometheus:prometheus

restart: always

################################################

alertmanager:

image: prom/alertmanager

ports:

- 9093:9093

volumes:

- ./alertmanager/:/etc/alertmanager/

networks:

- back-tier

restart: always

command:

- '--config.file=/etc/alertmanager/config.yml'

- '--storage.path=/alertmanager'

# alertmanager-sns-forwarder:

# image: datareply/alertmanager-sns-forwarder:0.1

# ports:

# - 9087:9087

#volumes:

# - ./alertmanager/:/etc/alertmanager/

#networks:

# - back-tier

#restart: always

#command:

# - '--config.file=/etc/alertmanager/config.yml'

# - '--storage.path=/alertmanager'

sachet:

image: sachet/pysa_debg_docker

ports:

- 9876:9876

volumes:#

- ./sachet/sachet.yml:/config/sachet.yml

networks:

- back-tier

restart: always

command:

# - '--config.file=/etc/sachet/notifications.tmpl'

- '--config.file=/config/sachet.yml'

- '--storage.path=/sachet'

cadvisor:

image: google/cadvisor

volumes:

- /:/rootfs:ro

- /var/run:/var/run:rw

- /sys:/sys:ro

- /var/lib/docker/:/var/lib/docker:ro

ports:

- 8080:8080

networks:

- back-tier

restart: always

deploy:

mode: global

volumes:

prometheus_data: {}

grafana_data: {}

networks:

back-tier:

front-tier:

Could you please help me to write alertmanager.conf, docker-compose.yml and sachet.conf or any other additional alert rule/file needed for SMS.

I am getting email alerts(whatever the alerts written in alert.rules), the same way I want SMS alerts as well.

Your help would be appreciated.

Thank you.

Simon Pasquier

unread,

Nov 26, 2019, 9:43:01 AM11/26/19

to mohd, Prometheus Users

Not using sachet and SNS myself, I'm not sure to be helpful.
From what I can tell, the "team-sms" receiver isn't used by any
Alertmanager route. If you want to receive emails *and* SMS for every
alert, modify the Alertmanager configuration accordingly first:

receivers:
- name: email-me
email_configs:
- to: rece...@gmail.com
from: sen...@gmail.com
smarthost: smtp.gmail.com:587
auth_username: "sen...@gmail.com"
auth_identity: "sen...@gmail.com"
auth_password: "password"

webhook_configs:
- url: 'http://10.10.10.xx:9876/alert'
to:
- '+919123456789'
from: '+919876543211'

Then look at Alertmanager and sachet logs if anything abnormal happens.

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a97b71c4-a7fb-4786-9bd1-499a6713205b%40googlegroups.com.

Reply all

Reply to author

Forward