How to (correctly) upsize a HA prometheus setup?

[Holger Freyther]

tl;dr: How to turn-up new Prometheus instances and not send (or silence) false alerts?

My HA setup consists out of running several independent instances that scrape, evaluate the recording rules and alert. That should be pretty standard and works great until I need to increase the retention size (or turn-up in a new cluster, maintenance, ..). To upsize I get compute and new storage and spawn a new instance of Prometheus. 

My troubles start with alerts using larger windows on increase/rate (e.g. check progress of a job over a longer period of time). The new instance has an empty TSDB and just started to scrape it is likely to trigger this alert.

I wondered if I can silence but AFAIK we make the label sets match across all instances of the HA set-up to allow Alertmanager to dedup.

Is there a best practice for this?

holger

[sayf eddine Hammemi]

Hello,
Doesn't it mean that your alert expression is not reliable enough? maybe you are trying to solve an XY problem here. did you try to make an expression that does not trigger without enough history?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c9d8c05d-ec07-4adc-b4bf-10cd88b0b5cen%40googlegroups.com.