Removing old data it not happening if "no space left on device" regardless retention params

[Shox]

We are experiencing a catch22, when there is "no space left on device" on attempt to write to /wal and disk space is not freed. After some investigation, it looks like removing old data is happening only after compaction, but compaction can't happen as no pace left on device. 

Is that a expected behavior? Is there any solutions around this? Or any ongoing work to prevent this?

I actually do not understand a reason why that was done by this way. Could somebody please help?

[Christian Hoffmann]

Hi,

I would say this is expected behavior and you would be expected to leave
enough space for Prometheus to breathe. As far as I understand,
compaction works by reading the existing blocks, merging them and
writing the new files to disk. The original files can only be safely
deleted when the new file has been written.

There is ongoing work regarding compaction [2], but I think it's related
to those cases where it is explcitly triggered via clean_tombstones.

[1] https://prometheus.io/docs/prometheus/latest/storage/
[2] https://github.com/prometheus/prometheus/issues/7957

Kind regards,
Christian