How to clean up data older X days in Prometheus ? - Retention policy is working only for the data which gets accumulated newly

172 views

Skip to first unread message

Rajesh Reddy Nachireddi

unread,

May 20, 2021, 5:00:46 AM5/20/21

to Aliaksandr Valialkin, Prometheus Users, brian....@robustperception.io, Julius Volz

Hi,

We have running prometheus instances in staging and production which have retention policy set to 100GB. we realised that it is more than the Disk(PVC) allocated for it.

How to delete data which is older than X days using command line or API since when we changed the retention period or size which is not deleting older chunks.

Could you suggest a way to change the retention policy during runtime and make sure it enforces that rule for older data as well. Manual deletion of chunks is leading to WAL replay issues since they have those references. Any suggestions would really help

Thanks,

Rajesh

Rajesh Reddy Nachireddi

unread,

May 25, 2021, 12:52:54 AM5/25/21

to Aliaksandr Valialkin, Prometheus Users, brian....@robustperception.io, Julius Volz