Reg: storage retention not working.

[RAJESH DASARI]

Hi ,

i am running prometheus with this option
--storage.tsdb.retention.time=24h but i see that storage space is not
cleared for every 24hrs. I see that data is stored from months and
disk is getting full and prometheus service is not starting due to
this reason.

prometheus[30086]: level=error ts=2021-01-24T04:04:00.908Z
caller=main.go:740 err="opening storage failed: open
/opt/pm/prometheus/wal/00131056: no space left on device"

Could someone please help, why this can happen and how to avoid this.

This issue is seen in one instance and we had prometheus in multiple
instances. Prometheus version used was 0.18.1

Thanks,
Rajesh Dasari.

[Bjoern Rabenstein]

On 04.02.21 21:55, RAJESH DASARI wrote:
>
> i am running prometheus with this option
> --storage.tsdb.retention.time=24h but i see that storage space is not
> cleared for every 24hrs. I see that data is stored from months and
> disk is getting full and prometheus service is not starting due to
> this reason.

This should just work. Perhaps your storage is already corrupted from
previous disk overflows so that Prometheus cannot delete the older
blocks?

What happens if you start from an empty storage directory?

> This issue is seen in one instance and we had prometheus in multiple
> instances. Prometheus version used was 0.18.1

That's a veeeeerrrry old Prometheus version (that, IIRC, doesn't even
have the --storage.tsdb.retention.time flag).

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in