Experimental SST partitioning feature

[Christopher Kujawa]

Hey everyone,

I have a question regarding the experimental sst partitioning feature. I stumbled over the here https://groups.google.com/g/rocksdb/c/l3CzFD4YBYQ and when trying it out in our environment it gave us really great results! But we are a bit unsure about the implications, and it is also not 100% clear how stable it is.

1. What implication does it have if we split up the SST files based on certain prefixes? We see much better performance in seeking/iterating, accessing data. Does this make sense?
2. What does experimental actually mean? What impact this might have? Will RocksDB drop the support for such features?
3. What are the guarantees versus normal features?

Happy to get some insights on this.

Greets
Chris

[Brian Munz]

From the Speedb Hive:

Breaking files by prefix is a good idea in some environments (for example MyRocks that uses prefix to define tables) . The iterator gain is only when you seek for all the keys in this prefix. The other gains may also be in compaction if you write multiple sequences   (one for each prefix)  The only pitfall is to ensure you do not create too many small files.  We have tested this feature and did not found any issue with it,,,

You can find the Speedb hive here and (once you've registered) the link to the thread here if you have more questions or need additional info

[Brian Munz]

Another comment from there:

Typically, experimental features are marked as such meaning that the API is subject to change.  Looking back through the history file, I believe this class was introduced in 6.12 in July 2020.  Since that time I cannot see any changes to this class.  YMMV

[Christopher Kujawa]

Hey thanks for your input!

I have a follow-up question regarding the SST partitioning. How would it behave or is it safe to turn the feature on and off? I experimented a bit with it and it seems to work.

What I mean is, what if I create a RocksDB instance (without a flag) store some data (maybe because we already have one). We now enable the flag, can I still access my data? From previous tests it looks like it, are there any things we need to consider?

Also in the other direction, we have the flag enabled but want to disable it again, will this hurt? I might not understand fully what the SST partitioning does, sorry for the dump question.

Greets

Chris

[Christopher Kujawa]

Just for posterity, I got confirmation via E-Mail:

> Partitioner is just a hinter when compaction should split the file. Default compaction is also splitting by file size. So it has no functional effect and you can change configuration anytime.
>
> Partitioner does not need to be simple prefix only, but one can use more complicated strategy.

> To me it saves a lot of compaction writes in case one use prefix as table identifier.

Thanks all.