Add metric label based on Kubernetes Namespace metadata

[Joshua Branham]

Anyone know of a way to add a label to all metrics that is the value of a label on the Kubernetes namespace of the incoming alert? for example, setting a label 'team_name' automatically for the metric kube_pod_container_status_restarts_total based on the value of 'team_name' on the namespace metadata. The goal is to find some way to dynamically decide what receiver to send an alert to based on metadata from k8s rather than explicitly defining it in the alertmanager config.

[Brian Candler]

If you have one timeseries kube_pod_container_status_restarts_total, and another timeseries which has a 'team_name' label, and there is some common label between the two (pod id?), then you can join them in your alerting rule expr so that the PromQL result includes the 'team_name' label.

For more info:

https://www.robustperception.io/how-to-have-labels-for-machine-roles

https://www.robustperception.io/exposing-the-software-version-to-prometheus

https://www.robustperception.io/left-joins-in-promql

https://prometheus.io/docs/prometheus/latest/querying/operators/#many-to-one-and-one-to-many-vector-matches

[Joshua Branham]

Thanks for the info, yeah I was hoping to do this all on the alertmanager side if I could. We use a lot of rules from this upstream and I was hoping to not have to modify them just to get this to work https://github.com/kubernetes-monitoring/kubernetes-mixin