Can service specific alert routing be defined with service deployment definition

13 views
Skip to first unread message

Vikrant Chaudhary

unread,
Jun 4, 2020, 9:57:32 AM6/4/20
to Prometheus Users

This may be a basic use case but I couldn't figure out based on the documentation I looked at. 


I am expecting  each of my service to create some service level alerts and package them as part of their helm charts which seems simple enough to do - each service creates a Prometheus rule for alerts it cares about. 


Here is my question - Is it possible for the service to allow implementing some of the routing rules as part of its deployment definition ? 


To me it seems like there is one config file where alert manager picks the routing information from. How can individual services update that config file (basically a configmap in a secret in k8s) as part of their deployment definition ?

Stuart Clark

unread,
Jun 4, 2020, 10:44:58 AM6/4/20
to Vikrant Chaudhary, Prometheus Users
You wouldn't generally expect Alertmanager routing rules to be services
specific, and equally are likely to vary depending on the setup of the
end user - they might choose different things to send to Slack or might
use Teams instead, etc.

You'd normally have some labels/annotations set by the alert detailing
some sort of severity, then the routing rules would match on those and
determin where an alert is sent.

--
Stuart Clark

Vikrant Chaudhary

unread,
Jun 4, 2020, 1:05:35 PM6/4/20
to Stuart Clark, Prometheus Users
Cool. Thanks for responding. Ya I think that makes sense. Just wanted to make sure its the expected work flow. 
Reply all
Reply to author
Forward
0 new messages