[ANN] Khepri 0.3.0 - A tree-like replicated on-disk database library

[Jean-Sébastien Pédron]

Hi!

On behalf of the RabbitMQ team, I'm pleased to announce Khepri 0.3.0! It
is available on:
* GitHub (https://github.com/rabbitmq/khepri/releases/tag/v0.3.0)
* Hex.pm (https://hex.pm/packages/khepri)

Khepri is a tree-like replicated on-disk database library for Erlang and
Elixir. This is a library we plan to use inside RabbitMQ as a
replacement for Mnesia. Indeed, we want to leverage Ra[1], our
implementation of the Raft consensus algorithm, to have a uniform
behavior, especially w.r.t. network partitions, across all types of data
managed by RabbitMQ.

The release notes [2] describe the content of Khepri 0.3.0 in details,
with examples. The most important and breaking change is the revamp of
the public API. Here is a summary of the major highlights of this change:

* The high- vs low-level API distinction is now gone. The public API is
now exposed by `khepri` only. `khepri_machine` becomes an internal
private module. As part of that `khepri` grew several new functions
for common use cases and we will certainly add more in the future,
based on the feedback.

* Unix-like path are first-class citizen: all functions taking a native
path ([stock, wood, <<"oak">>]) now accept Unix-like paths
("/:stock/:wood/oak"). In the process, the syntax of Unix-like paths
evolved: atoms are represented as :atom and binaries are represented
as-is, binary. The main reason is that using <<binary>> for binaries
was difficult to read and type.

* Payload and event filter records are now private. Payload types are
automatically detected now, likewise for event filters. That said, it
is still possible to use functions to construct the internal
structures, but it should rarely be necessary.

* `khepri_tx`, the module to perform Khepri calls inside transactions,
will now expose the same API as `khepri`, except when functions don’t
make sense in a transaction.

As always, I would loooove to hear from anyone who glanced at the code,
the documentation or even started to play with Khepri :-)

[1] https://github.com/rabbitmq/ra
[2] https://github.com/rabbitmq/khepri/releases/tag/v0.3.0

--
Jean-Sébastien Pédron

[Frank Muller]

Congratulations for the new release.

Did you update the benchmark against Mnesia? Is Kephri 0.3.0 faster?

[Jean-Sébastien Pédron]

On 27/04/2022 05:14, Frank Muller wrote:
> Congratulations for the new release.

Thank you!

> Did you update the benchmark against Mnesia?

Michael David, a contributor, did update the benchmark a few days ago.

> Is Kephri 0.3.0 faster?

No, nothing was done to improve the performance.

The throughput or latency are really dependent on the Ra/Raft cluster
underneath anyway. The write throughput increases with more concurrent
updates to a Khepri store. The reason is that fsync(2) calls are less
frequent compared to the number of Raft commands. fsync(2) is what
impacts the most the performance of Khepri. This is mandatory to provide
the guarantees of Raft. AFAIK, Mnesia doesn't perform any fsync(2)
automatically.

--
Jean-Sébastien Pédron