(Prometheus 2) storage.tsdb.max-block-duration when setting retention to 60 days

[Danny Kulchinsky]

Hey :)

We would like to increase the retention of Prometheus (ver 2) to 60 days, I wonder if the default setting of storage.tsdb.max-block-duration (10% of retention duration) is a good/sane default in this case?

Any suggestions?

Thanks!

Danny

[Brian Brazil]

It's a sane setting, and it's only a flag to allow for our own benchmarking for development purposes.

 

 

--

Brian Brazil

www.robustperception.io

[Danny Kulchinsky]

Understood :)

Thanks Brian!

[Ben Kochie]

I've found that when using long retention, say 1 year, 10% is too much.  We hit problems with very large indexes and failing compactions.  The index bug is fixed, but I found that 3-5% was better.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAHJKeLr3sAbLuUsOuOrijipVLCdDSndkgLgTnpt7SfSud4tE1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Danny Kulchinsky]

Thanks Ben, I would assume that for a 60d retention we shouldn't have the need to change the default value as Brian suggested, right?

Danny

On 1 February 2018 at 14:40, Ben Kochie <sup...@gmail.com> wrote:

I've found that when using long retention, say 1 year, 10% is too much.  We hit problems with very large indexes and failing compactions.  The index bug is fixed, but I found that 3-5% was better.

[Ben Kochie]

Correct, for 60d, the default is fine.

[Matt Doughty]

Out of curiosity, would the default still be sane with 90d retention?

--Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrbLc97wtJMkhELuJV7p03jpJC%3DVSE9oDJG9Dg3woghpA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

--Matt

[Ben Kochie]

It's probably sane up to 180d.  Then you're hitting over 2 weeks in the maximum block size.  Depending on label churn, the indexes could get very large.  For our 365d retention, we locked it to 7d to avoid overly large compactions.  It does mean the server has to open up more index files when doing longer queries, but those tend to be rare.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGyBzcwmK9QN5hi8f1BgvvJRDT_F9DpfmfvDsvjGmUkPzPq_Qw%40mail.gmail.com.