Empty activeAlertmanagers

[Anzor Baysarov]

Empty activeAlertmanagers list after executing reload method (0.001...5 sec)

1 Run prometheus with params (go run cmd/prometheus/main.go --config.file=config.yaml --web.enable-lifecycle) config.file:

YAML

1 2 3 4 5 6

alerting: alertmanagers: - http_sd_configs: - refresh_interval: 20s url: http://xxx timeout: 1m

2 make request curl -sXPOST http://localhost:9090/-/reload 3 and then you can try to open http://localhost:9090/api/v1/alertmanagers 4 Output will be empty. looks like:

{ "status": "success", "data": { "activeAlertmanagers": [], "droppedAlertmanagers": [] } }

5 Only after executing peace of code prometheus will update the content of config refs: https://gitlab.ozon.ru/sre/prometheus/-/blob/9619d3fd3b05122d2fad9d80391b100f976d3764/notifier/notifier.go#L309 ^ https://gitlab.ozon.ru/sre/prometheus/-/blob/432005826d06e6d6ac8f2d3836f7fdf66d9c3092/discovery/legacymanager/manager.go#L243 ^ https://gitlab.ozon.ru/sre/prometheus/-/blob/432005826d06e6d6ac8f2d3836f7fdf66d9c3092/discovery/legacymanager/manager.go#L97

as a result, prometheus has a time window from 0.0001 to 5 seconds when downtime is possible. How do you cope with such a task?