UKV for LevelDB: Arrow, Documents, Graphs, RPC, Python, and a new Interface Proposal

[Ashot Vardanian]

Hello, everybody!

My team is working on a different Key-Value Store, but we have recently open-sourced something potentially valuable to the LevelDB community. It is called UKV and is a shared interface for LevelDB, RocksDB, and a few other KVS.

Once you standardize something, it becomes much easier to build an ecosystem around it. So on top of the binary strings abstraction, we have built Document collections, Graph collections, and Vector indexes (in progress), similar to RedisJSON, RedisGraph, and RediSearch, bringing much broader functionality to LevelDB lovers. Document/tabular collections, for instance, feature both JSON, BSON, and MessagePack native support, as well as JSON-Pointer, JSON-Patch, JSON-MergePatch, and other RFC standards.

Moreover, UKV comes with a client-server communication library built on top of Apache Arrow Flight RPC, a combination of gRPC and Arrow IPC. You can deploy LevelDB on a remote machine, keeping the same UKV interface. This year we will also work on Sharding and Replication. Feel free to join!

Moreover, as UKV is implemented in C, adding higher-level bindings is a piece of cake. We now have bindings for C++,  Python, basic Java, and GoLang, and we plan to start working on JS soon. All under one umbrella. Identical SDKs across different stores, persistent or volatile, consistent or not, sharded or singe-node. Our SDKs take some time to craft as we minimize data copies on the hot paths but provide the simplicity of famous Python libraries people love. As such, tabular collections mimic Pandas, and graph collections mimic NetworkX, yet work with orders of magnitude more data, often stored on a remote disk.

We would love to invite everyone to join the development!

Thank you for reading, and enjoy UKV!

Repository

Documentation