Iterlib - An iterator library
Arun Sharma
[ This is mainly related to the key value store related discussion from a few weeks ago on this list]
This is an open source library that seeks to make it easy to build higher level functionality on top of RocksDB's iterators. RocksDB iterators are untyped by default and implement a sorted map from strings to other strings. What if you wanted to implement a sorted map from <int, int> => double?
Today we’re releasing a basic building block - iterlib::dynamic, which defines what the iterators are iterating over. This dynamic is similar to folly::dynamic and other C++ libraries such as JSON for Modern C++[1]. However, unlike other dynamics which tend to be self-contained and own the memory backing them up, iterlib::dynamic allows you to refer to existing memory in RocksDB’s caches, enabling a zero-copy implementation of the volcano iterator model[2].
The most immediate benefit you’ll notice is that the dynamic consumes only 32 bytes on most platforms. gcc5 regressed it to 40 bytes, but we’re looking to roll it back to 32. You can also add more custom types (e.g. latitude and longitude for a location related application) as long as it fits within 32 bytes.
More details in the README and the unit tests. We hope to do the rest of the development in open source.
https://github.com/facebookincubator/iterlib
Questions/Comments: ite...@googlegroups.com, https://groups.google.com/forum/#!forum/iterlib
[1] https://github.com/nlohmann/json
[2] https://paperhub.s3.amazonaws.com/dace52a42c07f7f8348b08dc2b186061.pdf
Wouter van Oortmerssen
--
You received this message because you are subscribed to the Google Groups "FlatBuffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to flatbuffers+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Arun Sharma
> Thanks Arun.
> Can you say a few words as to your plans on how this relates to FlatBuffers,
> for those that are not aware of what you're working on?
deserialization when objects are sent over the network. This is an
attempt to extend the zero-copy techniques all the way to storage.
Because of the need to modify the buffer so it sorts as intended on
storage, it'll involve some bit twiddling and byte shuffling. Our
intention is to minimize it and make it as efficient as possible,
while storage can continue to use cheap comparators such as memcmp.
We're also excited about the schemaless work as well as the reflection
capabilities in flatbuffers, since they allow schematization in
key-value stores without having to update the binary that reads the
embedded key-value store every time a new schema is added.
-Arun