Recommendations for defrag.limit thresholds

[ad...@metabroadcast.com]

Hi,

We're currently experiencing infrequent but inordinately long defrag processes on our Nimrod datastore. The current defrag.limit is set to 50. The problem I think is where the balance lies, and how the defrag process behaves as the dataset grows.

Our current db file is 43GB, and as such the implication is that there's only an actual data set of ~21GB? With a much lower defrag.limit, this would potentially ensure more frequent but shorter defrags, I expect? Of course, the time spent on defragging will only grow longer as the amount of data we hold grows. We're currently seeing outages of over 10 hours when the defrag process is running (which has a frequency of about once a month), which is unacceptable in production. As would weekly 4 hour defrags. 

Are there plans to evaluate different storage backends for Nimrod, or ways to address the defrag in background (or is this a fundamental limitation of HSQLDB?)

Regards,

Adam Horwich

[Sergio Bossa]

Hi Adam,

The lower you put the defrag.limit, the more frequent and short
defrags you'll get.
Given your amount of data, I'd suggest trying with a limit in between
5 and 10, and see if your use case can cope with that.

That said, if your dataset keeps growing, and you either do a lot of
manual deletes or have several metrics with sampling configured,
you're obviously right: whatever defrag limit you set, you're sooner
or later (let's say at approximately 100GB in your case) bound to long
wait times.

I'll see if I can find a way to improve on this area, but the obvious
choice of changing datastore is not really an option as of now, as
moving to a distributed/scalable store would make history and
aggregate queries difficult (if possible at all).

Another option would be to employ different Nimrod instances (hence
different databases): would this work for you?

> --
> You received this message because you are subscribed to the Google Groups
> "nimrod-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nimrod-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Sergio Bossa
http://www.linkedin.com/in/sergiob