block size calculation behaves unintuitively when only using size based retention

[Dom Prittie]

Hi,

I have a Prometheus deployment for which we have only used size based retention, which we have set to 2.5T. We regularly see unexpectedly large blocks getting created, so that when we hit storage.tsdb.retention.size we see a huge drop in the metrics available for querying.

For instance we currently have a block which is 1.1T and contains data from Mon 1 Jun 01:00:00 BST 2020 to Sun 14 Jun 13:00:00 BST 2020, which is ~45% of our retention size!

I can see from the docs that "Compaction will create larger blocks up to 10% of the retention time, or 31 days, whichever is smaller". I expected this to mean 10% of whatever retention I set, but from what I have seen it looks like this means the max block size will always be 31 days if you have storage.tsdb.retention.time=0s and are just using size based retention.

We have elected to use sized based retention instead of time because we are in the process of onboarding application exporters so the number of days we can retain is decreasing frequently, but the amount of space we have available is not. In this situation what would be the best way to configure storage.tsdb.max-block-duration?

Thanks,

Dom

[Stuart Clark]

What is the exact issue you are seeing, as it sounds like the storage
usage is still within the 2.5T limit you set?

--
Stuart Clark

[Dom Prittie]

The issue is with the size of the blocks being created and how the block size calculation appears to behave differently for time based vs size based retention settings. Just before we hit our retention limit (2.5T) we can query ~30 days of metrics, when we hit the limit the oldest block is removed and the size of that block often uncomfortably large. The example block I gave in my first email is 1.1T which represents about 14 days of data, so when that block is the oldest and we hit our size based retention limit 14 days of data will disappear at once, because Prometheus removes whole blocks at a time.

The thing that is odd is that I expect the behaviour around retention limits to be the same regardless of whether you are using just time based retention, just size based retention, or a combination of both. If I had used a time based retention of 30 days (currently roughly equivalent to my size limit of 2.5T) then instead of losing 14 days worth of data when we hit the limit we would lose 3 days of data or ~ 10%.

We don't wish to use time based retention at the moment because the amount of samples we ingest daily is in flux. So how can I ensure we get reasonably sized blocks (blocks <= 10% of storage.tsdb.retention.size)?

This e-mail together with any attachments (the "Message") is confidential and may contain privileged information. If you are not the intended recipient or if you have received this e-mail in error, please notify the sender immediately and permanently delete this Message from your system. Do not copy, disclose or distribute the information contained in this Message.

Maven Investment Partners Ltd (No. 07511928), Maven Investment Partners US Ltd (No. 11494299), Maven Europe Ltd (No. 08966), Maven Derivatives Asia Limited (No.10361312) & Maven Securities Holding Ltd (No. 07505438) are registered as companies in England and Wales and their registered address is Level 3, 6 Bevis Marks, London EC3A 7BA, United Kingdom. The companies’ VAT No. is 135539016. Maven Asia (Hong Kong) Ltd (No. 2444041) is registered in Hong Kong and its registered address is 20/F, Tai Tung Building, 8 Fleming Road, Wan Chai, Hong Kong. Maven Europe Ltd is authorised and regulated by the Financial Conduct Authority (FRN:770542). Maven Asia (Hong Kong) Ltd is registered and regulated by the Securities and Futures Commission (CE No: BJF060).

[Dom Prittie]

FYI - it appears someone else has run into this issue before: https://github.com/prometheus/prometheus/issues/6857

[Brian Candler]

If you add --storage.tsdb.max-block-duration=1d does this solve your problem?