Max # of rules , Cost of each rule, available vs non-available (match vs zero match)

[Rajesh Reddy Nachireddi]

Hi,

1. What is the cost of a alert rule in promethes? Interms of memory or cpu cost- No.of timeseries matched or something more than that?

2. How many rules are too many for a single prometheus ?

3. How large single alert rule can be ?

4. What is the cost of evaluation of a alert rule for which there are no metrics available? Is it same as if the metric is available or negligible? For example, if there is a rule on device=ip1 cpu metric, and there are no device=ip1 metrics, what is the cost of that rule? 

5. What is best way to generate rules or relabel config for large enterprise setups ? - Do we have some opensource/commercial tools which has UI or API CRUD or Only through Config management tools ?

Regards,

Rajesh

[Ben Kochie]

On Fri, May 29, 2020 at 9:13 AM Rajesh Reddy Nachireddi <rajesh...@gmail.com> wrote:

Hi,

1. What is the cost of a alert rule in promethes? Interms of memory or cpu cost- No.of timeseries matched or something more than that?

Rules are queries, so it depends on the rule and how much data the rule evaluates.

 

2. How many rules are too many for a single prometheus ?

Depends on the size of the server and the size, complexity, and frequency of your evaluation cycle. It could be single digits, it could be hundreds of thousands.

3. How large single alert rule can be ?

As large as a single query can be executed.

 

4. What is the cost of evaluation of a alert rule for which there are no metrics available? Is it same as if the metric is available or negligible? For example, if there is a rule on device=ip1 cpu metric, and there are no device=ip1 metrics, what is the cost of that rule?

Rules that match nothing are cheap. The index matching is reasonably efficient.

 

5. What is best way to generate rules or relabel config for large enterprise setups ? - Do we have some opensource/commercial tools which has UI or API CRUD or Only through Config management tools ?

We use code to generate much of our alerting configuration. Metrics per service are categorized and standardized to create SLO-driven data and alerts for all services.

See: https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog

 

Regards,

Rajesh

[Rajesh Reddy Nachireddi]

Thank you for the clarification.

Ben,

Do we have any tool or to benchmark the alerting part on prometheus?

to know how many series are fetched to Memory and also when the evaluation frequency is set to 10seconds then will it have caching of previous results and only the diff will get into memory calculation or entire thing will be addition because of too less eval interval ?

Does Cortex, Thanos, Victoriametrics support Rule or relabling via API or through UI ?

Thanks,

Rajesh

[Rajesh Reddy Nachireddi]

can someone reply for this