CRUD application backed only by immutable facts

[Kevin Lynagh]

Has anyone seen or implemented a CRUD application in Clojure using a
database of immutable facts?

For instance, a traditional database table supporting a todo-list
application has columns

user_id, task_id, task_description, is_done

A new row is created when a user adds a task.
Then that row is updated so is_done = TRUE when the user checks the
task off.

With immutable facts this would instead be a collection of statements:

User U added task T with description D at time T1
User U completed task T at time T2

To get a list of unfinished tasks for a user, you'd need to grab all
the tasks from this "transaction log", put them into a data structure,
and then remove ones when you learn that they've been completed.
Whatever is left over is the todo list.

Nathan Marz talked about this in terms of big data:

http://nathanmarz.com/blog/how-to-beat-the-cap-theorem.html

and Datomic's big bet is that your life as a developer gets much
easier when you just deal with (entity, attribute, value) + time.

I buy it in theory, but I have no idea what to expect in terms of
performance (e.g., how long would it take to find the current todo
list of someone who has added and completed/removed a few thousand
items?).

Has anyone implemented this idea on Clojure datastructures using,
say, (timestamp, keyseq, value) and reducing a ton of calls to assoc-
in?
Aside from speed, what are some other tradeoffs of an immutable
approach?

[Jonas]

On Tuesday, June 5, 2012 3:59:39 AM UTC+3, Kevin Lynagh wrote:

Has anyone seen or implemented a CRUD application in Clojure using a
database of immutable facts?

For instance, a traditional database table supporting a todo-list
application has columns

    user_id, task_id, task_description, is_done

A new row is created when a user adds a task.
Then that row is updated so is_done = TRUE when the user checks the
task off.

With immutable facts this would instead be a collection of statements:

User U added task T with description D at time T1
User U completed task T at time T2

I think this would be a great fit for datomic. Consider the following transaction:

[[;add entity :task/description "Make coffee"]

 [:add entity :task/deadline #inst ...]

 [:add entity :task/user user-entitiy]]

In order to mark this todo item as "done" simply retract the entity:

[[:retractEntity entity]]

In addition, the :task/user attribute could be of type :db.cardinality/many so many users would be assigned to the same task. I bet it would also be possible to make a todo-item depend on another todo item and ask smart (declarative) queries like "What tasks has to be done before this task and who should do them?"

[nicola...@gmail.com]

It is not totally clear in your post how you want to keep the data?
Is it in memory (with a transactional log somewhere)?
If it is the case, you can do better than reducing the whole data set
when executing a query:
you can keep a cache of query results, or indexed data and maintain
it, while still being
purely functional. (For example by attaching those results as meta
data to the data structure, and
defining your own assoc-like functions that maintain a consistency of
the meta-dataed query results)

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en



--
Sent from an IBM Model M, 15 August 1989.

[Kevin Lynagh]

Jonas,

Definitely inspired by the ideas in Datomic.
My question was partially: "how can I implement the core immutability
semantics of Datomic in plain Clojure?".
(Say, hypothetically, that I need a Clojure datastore with flexible
schema and immutability semantics but cannot actually use Datomic for
business reasons.)

I gave it a shot this morning---writing code to serialize

{:a 1 :b 2}

into

[#Assertion{:id 1, :attribute :a, :value 1, :time 0},
#Assertion{:id 1, :attribute :b, :value 2, :time 0}]

and back again (which gets a bit more complex with references/
collections, but not much).

Nicolas,

Data can be kept anywhere that tuples can be kept = )
Caching intermediate values "diffs are collapsed" should definitely
bring a speedup.
The immutability semantics mean you really can go all out crazy with
the caching too.



On Jun 5, 1:44 am, "nicolas.o...@gmail.com" <nicolas.o...@gmail.com>
wrote:

[Dave Sann]

This sounds a lot like Event Sourcing to me. (often coupled with CQRS).

Which, I think, is similar to what datomic is doing, where:

 - datom      => event

 - transactor => event store

 - peer         => query model

There are a number of presentations, blogs available on this if you haven't seen it previously. just Google for it

It sounds simple in theory - and I like the idea a lot in principle - but if you look at discussion boards, there seem to be challenges in actually applying it effectively. Management and Synchronisation of separate write and read model being significant.

Dave

> > clojure+unsubscribe@googlegroups.com

[Angel Java Lopez]

Yes, I thought the same... Now, encouraged by Dave message ;-) some links:

http://martinfowler.com/eaaDev/EventSourcing.html  

Dave mentioned CQRS, some post about Events, Event Sourcing AND CQRS:

http://thinkbeforecoding.com/tag/CQRS

CQRS (Command Query Responsibility Separation) is a big topic, possibly not needed for this thread. Some tutorial, intro links I collected

http://ajlopez.wordpress.com/2012/03/12/cqrs-links-news-and-resources-3-2/

clojure+u...@googlegroups.com

[江海龙]

I've been concerning this problem for a few months.

The "transaction logs" is actually the history events of an entity.
With an initial state is offer, the current state of an entity can be
calculated with the help of history events.
So if you want to know a property of an entity currently, these
calculation is inevitable.
But we can reduce the calculation process by taking snapshots for a
given entity.
A snapshot is another state of an entity at a specific time. You can
calculate the current state of an entity by just applying all the
later history events to the snapshot.

For example, suppose we have a task T, and it's empty map at start.
(def T {})

We can do push/pop operation on it with a history event generated. S
doesn't change at all, just some events are recorded.
(assoc T :title"A task")
(assoc T :time "today")
(dissoc T :time)
(assoc T :status "done")

So what is the current state of T?
(calculate-current T)
=> {:title "A task" :status "done"}
This state is generated by applying all the operations on the initial
state of T.

However, if there are a large amount of history events, a long time
will take to get the current state of S.
To improve this, snapshot is needed.
(assoc T :title"A task")
(assoc T :time "today")
(def T1 (calculate-current T))
(dissoc T :time)
(assoc T :status "done")

So what's the current state of T?
As we know, the last snapshot is T1, we can just apply (dissoc T1
:time) and (assoc T1 :status "done") to get the current state of T.
With the help of the snapshot, we save 50% of the calculation.
Of course, we need to use some more space to store the snapshots.

More consideration is needed if you want to query the state of an
entity at a specific time in history.
But I think it's enough to solve your problem, because you always need
the current state of a task. Thus you can just store one snapshot.

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en

--
Sincerely,
江海龙
Hoiloong Kong