ANN: Durable Clojure - Phase II - Persistent Data Structures

[Alyssa Kwan]

Extension of http://groups.google.com/group/clojure/browse_thread/thread/561230749be02f28

Hi everyone!

I've continued hacking the Clojure core to extend durability.

1. Durable atoms and agents are now supported in addition to refs,
using (datom) and (dagent) respectively.
2. Keywords and symbols are now supported.
3. Persistent data structures (except for tree-map and tree-set) are
now supported. Durable persistent data structures are truly
persistent: writing one with shared structure skips the part that has
been written before. This leads to an enormous performance bump, one
that I'm confident beats serialization schemes that are less
integrated with the Clojure core.

Get it here: git://github.com/kwanalyssa/clojure.git

TODO:
1. Support for refs of refs, i.e. a ref pointing to a collection of
refs.
2. Support for closures and functions. Help greatly appreciated!
3. Support for tree-map and tree-set. This requires functions because
of the comparator.
4. Lazy loading and unloading of persistent data structures. This is
probably orthogonal to durability for identities, but can share a lot
of the same architecture.
5. More unit tests! Especially for concurrent writes to identities!
6. Performance benchmarks!
7. Airtight ACID for drefs, which requires 2PC for in-memory refs.
This is a fundamental redesign of LockingTransaction.
8. Hygienic namespaces. This is a tough one.
9. Change API? Maybe use (ref {:durable true} …) instead of (dref
…). This may be a way to make the store arg optional with dynamic
bindings. Feedback please!
10. A way to maintain namespace identity without relying on a static
cache to get reference identity. I really don't like that I punted on
this implementation point, but a better way escapes me. Help on this
is REALLY welcome!

Feedback appreciated and patches welcome!
Alyssa Kwan