Federating Prometheus Rules

[Pieter Vincken]

Hi all

We're using the Prometheus operator to create Prometheus federates in all of our K8S cluster. We centralized a "master" Prometheus which has an Alertmanager connected to it and manages all the outgoing alerting. Currently we, the platform team, manage all alerting rules centrally in this master Prometheus. We'd like to move this power to the developers so they can create their own alerts with the PrometheusRule CRDs in the clusters. 

Is there currently a way to federate the rules from a "federate" instance to a master instance, like with the metrics? If not, what would be the desired way to properly set this up be? Deploy the alert managers as many times as we have federates? 

Thank you in advance.

Kind regards

Pieter

[Julius Volz]

In a typical setup you'd still only have one Alertmanager (or AM cluster), but the Prometheus servers for the different teams would all have their alerting rules evaluated locally and send their alerts directly to that central Alertmanager. Evaluating alerts in a global federated layer is mostly only a good idea when you need to take global context into account for your alerts (like "2 out of 5 clusters are down"), otherwise we suggest generating alerts as close to your targets as possible.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/68231942-4b48-420a-825e-5b40fe1be8dc%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Tobias Laving]

Potential hijack

What about recording rules?
We have a situation where team A provide a Prometheus instance with recording rules for kpis, team B want to scrape those kpis.
Team B could copy the kpis, but that could be error prone if the kpis are changed often.
What is the best way to resolve this?

Br,
Tobias

[Tobias Laving]

Nvm, we can get those metrics via federation. Due to faulty filter they was not included.