ANN: durable-persistence for Clojure and ClojureScript

[Christian Weilbach]

Hi,

I have started some explorations for durable persitent indices in a
cross-platform fashion, that I wanted to for quite some time now. There
are some other implementations of persistent datastructures with
synchronous IO on the JVM, but I would like to go a step further and
generalize over any key-value store like medium. I am primarily
interested in critique and references to other work or experience
implementing efficient indices on disk. In the longer run I want to use
these indices as a building block for databases in Clojure(Script).

https://github.com/replikativ/durable-persistence

From the README:

This repository is for exploration of persistent datastructures on a
durable medium. The durable medium is abstracted away by a minimalistic
key-value protocol providing atomic operations (ACID) on single keys.
This decouples the implementation of the persistent datastructure from
the host and backend store.

The repository contains an adaptation of the cookbook in-memory version
of a red black tree. Instead of keeping all fragments in memory we
introduce a Reference type to fragments in the store and dynamically
load and store fragments. The root can be assigned to a fixed key,
modelling a durable identity similar to a Clojure atom.

The long term motivation is to provide efficient indices as a building
block for datatype management independent of JVM IO libraries spanning
also to ClojureScript and the Browser. With konserve an IndexedDB
storage backend for the Browser already exists. For this reason IO
should happen asynchronously through core.async in konserve. A durable
Index for DataScript would be nice :).


Feedback is appreciated :),
Christian

[dgrnbrg]

This sort of thing is also the goal of Datacrypt & Hitchhiker trees (https://github.com/datacrypt-project/hitchhiker-tree), which are designed to be highly optimized for I/O.

We've been making progress on the GC and adding new backends--it would be awesome to be able add a CLJS port too :)

Best,

David

[lvh]

On Sep 8, 2016, at 6:39 PM, dgrnbrg <dsg123...@gmail.com> wrote:

This sort of thing is also the goal of Datacrypt & Hitchhiker trees (https://github.com/datacrypt-project/hitchhiker-tree), which are designed to be highly optimized for I/O.

We've been making progress on the GC and adding new backends--it would be awesome to be able add a CLJS port too :)

That’s awesome! What’s datacrypt, and how can I contribute? As a cryptographer who’s been working on CRDTs and does Clojure, this seems right in my wheelhouse :)

lvh

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christian Weilbach]

On 09.09.2016 01:39, dgrnbrg wrote:
> This sort of thing is also the goal of Datacrypt & Hitchhiker trees
> (https://github.com/datacrypt-project/hitchhiker-tree), which are
> designed to be highly optimized for I/O.

Sorry for the late reply, I was a bit busy. I have also implemented a
redis backend for our async kv-store protocol:
https://github.com/replikativ/konserve/blob/master/src/konserve/carmine.clj

Thanks for pointing out, this is very nice! I have just read
https://github.com/datacrypt-project/hitchhiker-tree/blob/master/doc/hitchhiker.adoc
Using a B+ tree approach would also have been my next take, but the
hitchhiker combination with an append log is really cool. I am
interested in having these sorted datastructures as building blocks for
CRDT metadata in the longer run.

>
> We've been making progress on the GC and adding new backends--it would
> be awesome to be able add a CLJS port too :)

Cool :) I have only thought about GC so far. To support cljs one needs
to be aware that all blocking IO calls won't work. That is why I have
started with a core.async kv-protocol. Can you imagine using something
like it?

To support Datascript in cljs, one would then also need to use something
along the lines of async operations to fetch index fragments during the
query. But for the current hitchhiker tree it should not be too invasive
and the performance should not be affected. I think IResolve, IBackend
(and maybe some more), need to return go channels, the rest can be added
gradually. I haven't looked into it in detail yet.

Have you had a look at replikativ? I would be interested in feedback and
critique :).

I don't know what your current take on CRDTs is (I have found the nice
OR-SWOT in libdistsys :) ), I will be doing my master thesis with
Annette Bieniusa soon, one of the authors of the paper about it. I
haven't done as much Clojure backend integration as you yet, since I
wanted to have cljs support for an open replication system and couldn't
wire any JVM technology in (including the implementation of synchronous
standard datatype protocols for asynchronous operations). But I think it
is definitely a good way to start in the Clojure backend as you have
done. As far as I can see the CRDTs in libdistsys are supposed to be
used with a network (the simulator is a good idea). Have you plans for a
complete middleware?

Best,
Christian

P.S.: I have played around with piplin two years ago, do you still work
with FPGAs?