alertmanager max number of rules?

[Jerry Lee]

What's the guidance in terms of max number of alert rules in a single Promethues server? I guess it's up to the complexity of alert rules, volume of metric database and server capacity, but any experiences to deal with the high number of alert rules will be helpful. 

As of today, I have more than 15,000 rules and experiencing alert process delay around an hour. 

Thanks, 

[Ben Kochie]

If you have more than a couple hundred, you're probably doing something wrong. Rules should be generic, one alert is designed to cover all instances in a set.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/696e7c15-6bdb-410d-830f-1e61a45c689fn%40googlegroups.com.

[Jerry Lee]

I agree, we don’t do a good job governing the rule registration.  Keeping no of alert rules less than a thousand could be a golden rule? 

[Ben Kochie]

I'd say no more than 100 per Prometheus server is a good rule of thumb. One rule should cover every single instance of an alert. For example "CPU Saturation" is one alert rule, You should create that rule and never have it change.

https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/

[Jerry Lee]

Thanks for the tip!