prometheus server in crash loop when volume is full

[Laurent Demailly]

What did you do?
Setup using community helm chart (8Gi data volume, 15 days retention)

What did you expect to see?
Would run indefinitely without crashing

What did you see instead? Under which circumstances?
Entered crash loop after 13 days because the volume got full.

I expect instead some warning that older data is being deleted sooner than configured retention. or an alarm. Not a crash loop.

Environment
k8s

(https://github.com/prometheus/prometheus/issues/8159 apparently must be an faq or "not a bug")

[Laurent Dumont]

Your retention is 16 days and you filled the data volume in 13?

I don't Prometheus will alarm for it's data partition by default.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f2a6a48a-7ab7-4df7-a8cc-3c2126e5c5d9n%40googlegroups.com.

[Laurent Demailly]

I guess my suggestion is that crashing isn’t exactly a user friendly response to something that can happen days or weeks after initial setup (if only because of data growth) and I suggest prometheus would more actively try to prevent it by dynamically shrinking retention for instance

[Laurent Dumont]

I don't think Prometheus will handle time.

But you can probably use --storage.tsdb.retention.size and have it slightly lower than the size of the actual data volume. That way, you won't fill up the volume and crash the server.