Compaction frequency

[Jamie Johnson]

I am looking to use Journal.IO as a write ahead log that I would like to periodically compact, my question is what is a reasonable amount of time between compactions?  Also are there any statistics on how long compactions typically take to run?

[Sergio Bossa]

On Wed, Jul 10, 2013 at 2:32 AM, Jamie Johnson <jej...@gmail.com> wrote:

I am looking to use Journal.IO as a write ahead log that I would like to periodically compact, my question is what is a reasonable amount of time between compactions?  Also are there any statistics on how long compactions typically take to run?

It really depends on how much data you delete, and how much fragmented such deletes are: you pay the cost of sequentially scanning all data files, plus the rewriting of all valid entries, so the more you delete the faster it will be, and you can compact more frequently; if you do not delete much instead, I suggest to wait for some garbage to accumulate before running compaction.

Future versions may implement parallel compaction, or if you wanna have a try at it, pull requests are well accepted :)

Hope that helped by the way,

Cheers,

Sergio B.

--
Sergio Bossa
http://www.linkedin.com/in/sergiob

[Jamie Johnson]

Thanks Sergio.  I am expecting that will be doing a lot of deletes, the entries need only stay around until the items in the log have been committed to the datastore.  I was originally thinking about having a task that would periodically compact the journal but this sounds needless if there haven't been some deletes since the last compaction.  Do you have any thoughts about at what point it becomes effective to do the compaction?  I'm just trying to get a feel for at what point we should be doing the compaction if there are say "X" number of deletes since the last compaction for instance.