Modeling Data Associations in Clojure?

[Brenton]

I am starting to write a large web application using Clojure and
Compojure and am running into some design trouble while designing my
data model. To illustrate my problem I am going to make up some fake
data. Suppose you are writing an Insurance application which has the
tables Policy, Person and Vehicle. A policy has a person that is the
policy holder. A policy also has many vehicles. Each vehicle has a
person that is the primary driver. Most of the time the primary driver
is the same as the policy holder. If I were using one of the usual
object-relational mapping frameworks, (Hibernate, ActiveRecord) when I
load a policy I would get an object graph. If the person who is the
policy holder is the same as the person who is the primary driver of
the vehicle then the loaded Person object would be the same object. If
I change the address of the policy holder the primary driver's address
will also be changed.

How do people deal with this sort of thing in a Clojure application
(or any other functional language)? At first I thought that it would
be easy and I would just use nested maps. But this causes all kinds of
problems. If I load the data into nested maps I now have two distinct
maps for the same person. If I change one of them, the other is not
updated. If I try to save this map back to the database, which person
map has the correct data? It is also awkward to update the person in
the first place. In Java you would just go policy.getPolicyHolder
().setAddress("..."). In Clojure you would have to do something like
(assoc policy :holder (assoc (:holder policy) :address "...")).

I have a feeling that there is a more "functional" way to do this sort
of thing. My question is, how to other people deal with this? The only
thing that I can think of is that I would avoid using nested maps to
model the database associations. I would load the policy into a map.
Then if I need the person, I would load that into a separate map. That
may be the correct functional approach. I was just asking in case
there is some really cool thing that people do that I don't know
about.

I had a look at clj-record to see how associations where handled. It
looks like nested maps are avoided here. Instead functions are created
to retrieve the associated data. Is the correct way of this this?

Thank you,
Brenton

[Dragan Djuric]

I think that you made the mistake in the first step: you cannot
*change* a clojure's map. Thus, you cannot change one particular
"snapshot" of person's data. No problem up to this point. You can only
create a new map with some data reused. There lies the problem - how
to "update the references" to the new map at all the places that link
to it. My hunch tells me to use refs for that :)
As for any examples, I cannot point you to that. It seems to me that
this problem, in my opinion one of the central problems in "real-
world" programming, is suspiciously underinvestigated in the world of
FP...
I am currently developing a DSL for dealing with the same problem, but
as it is in the rather early stage of development, no code is released
to the public yet.

[Luc Prefontaine]

We mainly use macros to create functions to deal with associations,
a bit like ActiveRecord except that it is not yet as dynamic. We do not use the table meta data to find the associations
and create the finders, etc...
We want eventually to add stuff to clj-record to make it more like ActiveRecord in this regard. For now it's the urgency to deliver that is driving the schedule.
We also maintain a cache within these functions to speed up things. The cache is not yet distributed again because of the schedule.

At least with these macros we are kind of half way toward the final goal....

Luc

Luc Préfontaine Armageddon was yesterday, today we have a real problem...

[Andy Kish]

Hi Brenton,

Nested maps are a good way to start, but they're pretty low level as
you want to do more complicated things. If you're talking about data
associations, the relational model is higher level and it's really
worth modeling your data in that way.

Relational data manipulation doesn't require a sql database. If you
want to use an in memory data representation, clojure has a couple of
options. The clojure.set namespace [1] has some useful functions
operating on sets of maps, in a relational way. join, select, and
project will get you pretty far.

A more structured and powerful way of doing things is
clojure.contrib.datalog [2]. I haven't had a chance to play with this
(yet!), but it looks very cool. It's a functional approach and it's
more well integrated with the language than something generating sql.

If your app really does call for connection to an external db, then
there's no reason not to go with clj-record. You wouldn't be the first
person to make a SQL db backed web app. :) Using a database through
clojure feels a lot more natural than via an ORM.

I'm doing a small web app with compojure to familiarize myself with
clojure right now, so report back to the mailing list if something
works well for ya! I'm not at the point where I need a db, but if I do
I think I'll end up trying to use datalog.

Andy Kish.

[1] http://clojure.org/api#toc654
[2] http://richhickey.github.com/clojure-contrib/doc/datalog.html

[Stuart Sierra]

Hi Brenton,

I think the simplest solution to this problem is to use functions
instead of maps. That is, instead of defining your API in terms of
maps with specific keys, define it in terms of functions that read/
write individual fields.

For example, you would have an opaque "Person" object, perhaps in a
Ref, and functions like get-name, set-name, get-policy, etc. The
underlying storage model can be whatever you want -- sets, SQL,
files, .... You just have to train yourself to never touch a model
object except through its defined API.

-SS

[Dragan Djuric]

Ha, ha, some object-oriented lessons are being rediscovered :)))

[Stuart Sierra]

On Sep 15, 6:54 am, Dragan Djuric <draga...@gmail.com> wrote:
> Ha, ha, some object-oriented lessons are being rediscovered :)))

Precisely! Just because the language doesn't enforce information
hiding doesn't mean you can't do it.

-SS

[Brenton]

Thank you all for your input. I think I will follow Stuart's advice
and go with something like the following, again using the example data
above.

(use 'clojure.set)

(def policies (ref #{{:id 3 :name "x" :holder 7 :vehicle 11}
{:id 4 :name "y" :holder 2 :vehicle
12}}))

(def vehicles (ref #{{:id 11 :make "Audi" :driver 7}
{:id 12 :make "Toyota" :driver 2}}))

(def people (ref #{{:id 7 :name "Brenton"}
{:id 2 :name "John"}}))

(defn find-policy [id]
(first (select #(= (:id %) id) policies)))

(defn get-field [map key table]
(first (select #(= (:id %) (key map)) table)))

(defn get-holder [policy]
(get-field policy :holder people))

(defn get-driver [vehicle]
(get-field vehicle :driver people))

(defn get-vehicle [policy]
(get-field policy :vehicle vehicles))

This feels very natural to use

(def p (find-policy 3))
(-> p get-vehicle get-driver)

I may consider using clj-record since it just uses macros to create
these kinds of functions for you.

Thanks again,
Brenton

[patrickdlogan]

Insurance policies, etc. are complicated graphs even for a relational
database model. I am not convinced you have to use clojure's
functional data structures for these graphs. I've seen good ORM
frameworks run out of steam on insurance apps as well.

As part of the _processing_ of insurance functions, I would use the
functional structures for sure. But the full graphs (insurance
documents) themselves? Not so much.

After working with various insurance implementations, I currently
believe one of the best representations is a good graph database
(whether durable on disk or just in memory). Something like
AllegroGraph provides additional features like querying and inference
over v.large graphs.

It's real hard building insurance graph processing from scratch but
clojure can use the java API to AllegroGraph, and there are other good
open source graph/semantic systems as well.

[luskwater]

You might also look at "Out of the Tar Pit" (http://web.mac.com/
ben_moseley/frp/paper-v1_01.pdf) by Mosely and Marks. It's an
argument for the relational model that Stuart mentioned, with an
example. (Warning: the example is a few pages out of the 60-odd pages
of exposition and argument: great paper, though.) Reading it gave me
the impression, "I'm doing it completely wrong," in the sense of
allowing mutable data.

[patrickdlogan]

re: "clojure.contrib.datalog"

Also note that some graph databases like AllegroGraph (not to over
promote that specific product, but it is free to use up to 50M triples
or somesuch)...

Anyway, such graph databases often provide a Prolog or Prolog-ish
inferencing capability (among other algorithms). And these backward-
chaining engines are not unlike datalog's. So there is somewhat of a
migration path I would suspect from the clojure in-memory datalog
library up to a full-featured graph database. YMMV

[eyeris]

If you already have a data model mapped to an object model via
Hibernate (or similar Java-based product), you could simply wrap a
clojure API around those classes via the Java interop. However, I
don't know of an example of published code that does this.

[Dragan Djuric]

Yes, but then it's just Java semantics disguised in a lispy syntax.
What I really like in Clojure are immutable data structures,
multimethods, closures, macros etc.
If we have to use Hibernate and Spring and all that (which is not bad,
but still looks too complex and bloated from the perspective of FP)
then why bother with Clojure...
Don't get me wrong: I think that Java interop is the most important
feature of Clojure, but for calling libraries, not building on
frameworks.

[eyeris]

Like I said, this applies if you already have a data model mapped to
an object model via Hibernate. You can leverage that mapping for the
time being and write what code originally prompted you to take up
clojure. This is especially attractive when you are introducing
clojure into a production program that is in production.

Wrapping your java objects with a clojure API does not require you to
retain the semantics. It only requires you to translate them to
clojure, which I think would be easier than translating the relational
database semantics to clojure.

If you don't already have your data mapped to an object model, then
it's likely not worth the effort. In that case, look into one of the
other suggestions on this thread.

[Dragan Djuric]

You're right about that.

I commented to a general solution. The point is that the community has
to develop a set of patterns/practices/whatever for this use case
since almost any "ordinary" application software has to solve it.
For 101 questions like this, there has to be at least one or more
straight answers with code examples. At least if we want Clojure to be
picked up by the general Java community. Not everyone has to compute
Fibonacci's sequence but everyone has to support some customers buying
some products ;)