alertmanager: map labels to opsgenie tags

2,182 views
Skip to first unread message

David Karlsen

unread,
Mar 31, 2017, 6:05:56 PM3/31/17
to Prometheus Users
Hi.

Is it possible to map all labels (which may be unknown in advance - I want to map all of them generically) over to opsgenie tags?
That way we can easily let teams route alerts in opsgenie based on whatever labels they have defined.

TIA,
David

Julius Volz

unread,
Mar 31, 2017, 6:14:14 PM3/31/17
to David Karlsen, Prometheus Users
The "tags" field of the OpsGenie notification config is templateable and has access to the alert metadata, so this should be possible. OpsGenie tags are simple strings, not key=value pairs though. Or do you want to encode the k=v pair into a single string for each alert label? Also note that a notification can contain multiple alerts with different labels, so only the .CommonLabels and .CommonAnnotations should be used for this in the template.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/55f4992b-9d15-490d-aa27-f2e2d7071f65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

David Karlsen

unread,
Apr 1, 2017, 6:20:26 AM4/1/17
to Julius Volz, Prometheus Users
I think I'd like key=value - so that it's semantially richer in the opsgenie UI and also more informational for the one receiving the alert.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6Yoz6MLPR0WGYOsXPgucRO_th5MkqehLf44mf%2BAMLjvkQyw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--
--
David J. M. Karlsen - http://www.linkedin.com/in/davidkarlsen

David Karlsen

unread,
Apr 1, 2017, 12:07:40 PM4/1/17
to da...@davidkarlsen.com, Julius Volz, Prometheus Users
I tried adding tags: myStaticTag,{{.CommonAnnotations}} to the opsgenie config section of alertmanager - but tags get truncated

[d1, map[label1:labelvalue1 label2:someValueButGetItWillBeTruncated

so it seems to get truncated (the actual values are anonymized)

Julius Volz

unread,
Apr 1, 2017, 4:43:51 PM4/1/17
to David Karlsen, Prometheus Users
Try something like this (same for annotations):

{{ range $k, $v := .CommonLabels}}{{$k}}={{$v}},{{end}}

Note that you'll have a comma after the last element as well this way, plus it won't escape commas in label values for you.

Also, OpsGenie won't know about the k=v structure (because they just use single-string tags), but maybe it'll be helpful anyways.

David Karlsen

unread,
Apr 2, 2017, 5:24:02 AM4/2/17
to Julius Volz, Prometheus Users
Great - thanks a lot.
Actually there is a "details" element in opsgenie which fits labels like a glove as it's key/value based: https://www.opsgenie.com/docs/web-api/alert-api#createAlertRequest - maybe there could be an option for opsgenie integration "mapLabelsToDetails" to propagate them over here.
It has more space (8k) vs tags which is max 20, each max 50 chars

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BT6YoyLufCKN%3Dbun%2Bu5v0hbkSTP%2BjcLZ8Fmkfz3oESrCct8Gw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

Julius Volz

unread,
Apr 2, 2017, 7:37:38 AM4/2/17
to David Karlsen, Prometheus Users
Good point - maybe that should even be the default behavior. Enabling something like this at least would make sense to me.

Care to file an issue (of course, feel free to submit a PR as well)?

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGO7Ob3KNAQtVyKTFY7d24Dc3r8GLPQCNGFSqX8NpKttaRutmw%40mail.gmail.com.

David Karlsen

unread,
Apr 2, 2017, 7:41:46 AM4/2/17
to Julius Volz, David Karlsen, Prometheus Users
Will do. I actually see a details element in the opsgenie config on the alertmanager side of things so I'll play around a bit with that and come back with a proposal. 

Thanks again. 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
--



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

Julius Volz

unread,
Apr 2, 2017, 8:12:47 AM4/2/17
to David Karlsen, David Karlsen, Prometheus Users
Yeah, you can currently specify the Details field in the config, but you have to manually list each key/value pair there.

Thanks again. 


To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
--



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGO7Ob18THQk_ikkF_3y0KFxOpVZMshqw_sHywpyJKjtBvBuKA%40mail.gmail.com.

vikram yerneni

unread,
May 27, 2020, 5:04:23 PM5/27/20
to Prometheus Users
Hi David Karlsen,
How did you end up setting this requirement? I am trying to set the same  so that the alerts can be routed to teams by OpsGenie through tags or labels (trying to find the best way to achieve this).
Any tips will be helpful.

Thanks
Vikram
Thanks again. 


To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
--



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages