Prometheus not restoring Alert state on restart.

3 views
Skip to first unread message

soha...@gmail.com

unread,
Jun 26, 2020, 4:40:58 AM6/26/20
to Prometheus Users
Say an alert is in firing state then after a few minutes, I restart the Prometheus. After the restart, Prometheus sends the resolved notification to the alert-manager (and alert-manager to the webhook).

Is there any way to restore an alert state for example if it was in firing state than it should remain in firing state after a restart and not send resolved alert.
Note alert rule for=5m

Prometheus version: 2.17.2
Alertmanger version: .0.21.0
Env: running as stateful sets in Kubernetes

Prometheus runtime args:
rules.alert.for-grace-period=3m 
rules.alert.for-outage-tolerance=1h 
storage.tsdb.retention.time=2h 

Prometheus config:
global:  
scrape_interval: 30s  
scrape_timeout: 10s  
evaluation_interval: 30s 

Please let me know if need more information

Reply all
Reply to author
Forward
0 new messages