promise pipelining aware methods for consistent databases

[Smurf En Drek]

Hello!

I'm considering using Cap'n Proto for a consistent Key Value database in Rust. It's mainly the Promise Pipelining that caught my eye over other RPC frameworks, because that would allow me to provide support for more complex queries based on simple get and set methods rather than having to parse larger queries myself.

The database I'm aiming to create should be linearisable, so it wouldn't be acceptable that in a conceptually pipelined call like get(get(mycatname)+"_color) (composed operation to get the color of my cat) that mycatid is edited halfway trough the request and that the request returns the outdated cat color. To avoid this, I assume the methods should be made aware of them being pipelined so they can pass a set of locks on some keys around between method calls so that everything stays consistent, even with pipelined calls.

Are there Cap'nProto features that would help me with this?

[Ian Denhardt]

Quoting Smurf En Drek (2020-10-16 20:46:37)

> To avoid this, I assume the methods should be made aware of them being
> pipelined so they can pass a set of locks on some keys around between
> method calls so that everything stays consistent, even with pipelined
> calls. Are there Cap'nProto features that would help me with this?

Probably what I would do here would be to explicitly model transactions
in the interaces, e.g. you might have something like:

interface Database {
startReadOnlyTx @0 () -> (tx :ReadOnlyTx);
startReadWriteTx @1 () -> (tx :ReadWriteTx);
}

interface ReadOnlyTx {
get @0 (key :Data) -> (value :Data);
}

interface ReadWriteTx extends ReadOnlyTx {
set @0 (key :Data, value :Data);

commit @1 ();
# Commit the transaction. To roll back/abort, drop the transaction
# object without committing.
}

...you could then put the logic to acquire locks and whatnot in the
implementations of ReadOnlyTx/ReadWriteTx.

> get(get(mycatname)+"_color)

Note that promise pipelining itself only allows you to call methods and
pass values around; if you want to do something like string
concatenation you will have to either (1) suffer the round trip, or (2)
expose methods on the server for the relevant operations; the calculator
example may be of interest here for an example of how you might model
this:

https://github.com/capnproto/capnproto/blob/master/c%2B%2B/samples/calculator.capnp