Hi everyone,
my configuration is:
prometheus : 2.2.1
alertmanager: 0.15.0-rc.1
I have faced with some difficulties , my alertmanager doesn't send "endsAt" parameter in json with webhook.
Steps :
1. i have triggered alert (for example high cpu load on somenodename)
2. Prometheus sends to alertmanager alert
3. Alert manager send next json to telegram bot via webhook:
{
"alerts":[
{
"annotations":{
"resolve_message":"CPU usage \u003c 80% during 5 minutes",
"summary":"CPU usage is too high (crnt: 100%; condition: 80%; during 5 minutes)"
},
"sendsAt":"",
"generatorURL":"11111",
"labels":{
"alertname":"HighCPU",
"dashboard":"1111111",
"environment":"11111",
"monitor":"11111111",
"nodename":"somenodename",
"project":"1111111",
"severity":"warning"
},
"startsAt":"2018-05-29T13:09:45.392819306Z"
}
],
"commonAnnotations":{
"resolve_message":"CPU usage \u003c 80% during 5 minutes",
"summary":"CPU usage is too high (crnt: 100%; condition: 80%; during 5 minutes)"
},
"commonLabels":{
"alertname":"HighCPU",
"dashboard":"111111",
"environment":"11111",
"monitor":"1111111",
"nodename":"somenodename",
"project":"1111111111",
"severity":"warning"
},
"groupKey":0,
"groupLabels":{
"alertname":"HighCPU",
"dashboard":"1111111",
"nodename":"somenodename",
"project":"1111111"
},
"receiver":"telegram",
"status":"firing",
"version":0
}
4. Stop cpu load on
somenodename.5. Prometheus sends resolve
6. Alertmanager sends post json (without "EndsAt"):
{
"alerts":[
{
"annotations":{
"resolve_message":"CPU usage \u003c 80% during 5 minutes",
"summary":"CPU usage is too high (crnt: 100%; condition: 80%; during 5 minutes)"
},
"sendsAt":"",
"generatorURL":"11111",
"labels":{
"alertname":"HighCPU",
"dashboard":"11111",
"environment":"11111",
"monitor":"111111",
"nodename":"somenodename",
"project":"111111",
"severity":"warning"
},
"startsAt":"2018-05-29T13:09:45.392819306Z"
}
],
"commonAnnotations":{
"resolve_message":"CPU usage \u003c 80% during 5 minutes",
"summary":"CPU usage is too high (crnt: 100%; condition: 80%; during 5 minutes)"
},
"commonLabels":{
"alertname":"HighCPU",
"dashboard":"111111",
"environment":"11111",
"monitor":"11111",
"nodename":"somenodename",
"project":"11111111",
"severity":"warning"
},
"groupKey":0,
"groupLabels":{
"alertname":"HighCPU",
"dashboard":"111111",
"nodename":"somenodename",
"project":"111111"
},
"receiver":"telegram",
"status":"resolved",
"version":0
}
The main idea - i want to compare "EndsAt" and "StartsAt" and write in resolve message alert's active time , for example "alert was fired during 7 minutes".
"alerts": [
{
"labels": <object>,
"annotations": <object>,
"startsAt": "<rfc3339>",
"endsAt": "<rfc3339>"
My alertmanager config :
# Ansible managed
global:
resolve_timeout: 2m
receivers:
- name: telegram
webhook_configs:
- send_resolved: true
route:
group_by:
- alertname
- nodename
- project
- instance
- queue_name
- replication_status
- dashboard
- domain
- host
- queue
receiver: telegram
repeat_interval: 8736h
routes:
- continue: true
group_by:
- alertname
- nodename
- project
- instance
- queue_name
- replication_status
- dashboard
- domain
- host
- queue
match_re:
severity: ^(critical|warning)$
receiver: telegram
repeat_interval: 8736h
Probably I misunderstood alertmanager documentation ....
Sorry for my english it is not my native language.