Monitor Disk Space on Prometheus server itself

[Tzach Livyatan]

Hello

For my servers, I used node_exporter to export metrics to Prometheus, and alert on disk usage.

Should I do the same for the Prometheus server itself, or does Prometheus already export such a metric (is it prometheus_tsdb_storage_blocks_bytes?)

My motivation is alerting on Prometheus disk usage before it fails.

Regards

Tzach   

[Chris Siebenmann]

My view is that you should do this through node_exporter on the
Prometheus server, even if you can get similar metrics from Prometheus.
The node_exporter metrics are directly exposing what the filesystem
says, and it is what the filesystem says that matters the most; if
Prometheus thinks it's not out of space but the filesystem thinks it is,
the filesystem wins and you have a problem.

(The direct filesystem metrics will also include usage from other things,
not just Prometheus. Perhaps in theory there shouldn't be any other usage,
but in practice sometimes things happen, including surprise gigantic core
dumps.)

- cks

[Lars Schotte]

Yes, on older fashioned filesystems for example also the number of
Inodes. Because it is usually the case that you have kind of a feeling
for space usage. So you will have often done something before the
filesystem runs out of space. But it happened to me more often that I
was caught by surprise by running out of "space" which actually was
caused by used up inodes.

[Simon Pasquier]

Note that recent versions of Prometheus support size-based retention policy:
https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/41cac9cc-e084-4ace-a9b8-a67caa6a6546%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Tzach Livyatan]

Thanks, Chris

Make sense

 

        - cks