Avoiding periodic compaction on bottommost level

[Neethu Haneesha]

Leveled compaction DB.

I want to enable periodic compactions and I am not using any compaction filter. The main intention of using periodic compactions is have an upper limit to get back the space trapped in tombstones in non-bottommost levels.
For my use case, I want to run the periodic compaction on all non-bottom most level and not on bottommost level as it does not have any deletes.
So, is there any way to avoid periodic compactions on bottommost level?

[MARK CALLAGHAN]

I don't have a direct answer to your question. But old versions are unlikely to be dropped by compaction prior to reaching the max level of the LSM tree as I explain here. Although my focus in the blog post was on tombstones, and I don't know whether your workload uses Delete(). And by "unlikely" I mean for typical workloads, your workload might not be typical. 
https://smalldatum.blogspot.com/2020/01/deletes-are-fast-and-slow-in-lsm.html

This isn't an issue with SingleDelete tombstones
https://github.com/facebook/rocksdb/wiki/Single-Delete

I assume there is a pure-RocksDB tool that can be used to determine whether this is true for your workload by analyzing the content of your SST files, but I tend to use MyRocks and have less experience with the pure-RocksDB tools. The output from compaction IO statistics, which might be written periodically to the RocksDB log might be a good start. What is the value of the KeyDrop column per level of the LSM tree. If it is much larger 
https://github.com/facebook/rocksdb/wiki/Compaction-Stats-and-DB-Status

An example of compaction IO statistics is here -- https://gist.github.com/mdcallag/30f3ad454808d6cba8c95640f9de38b7
And note that both the absolute value for KeyDrop is largest for L7. Also, the ratio (KeyDrop / KeyIn) is very small for L0, L3, L4 and L5, gets larger for L6 and is largest for L7. The ratio is ~0.25 for L7, ~0.05 for L6, ~0.007 for L5, ~0.0019 for L4 and even small for L3 and L0. Note that L1 and L2 aren't there because dynamic leveled compaction is used.

--
You received this message because you are subscribed to the Google Groups "rocksdb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rocksdb+u...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rocksdb/842badcf-2c1c-46f4-b3d4-e0bb4d985737n%40googlegroups.com.

--

Mark Callaghan
mdca...@gmail.com