Persistence and Clojure (was: <cfpropery> vs Boilerplate Accessors, etc.

[Sean Corfield]

On Tue, Feb 5, 2013 at 6:34 AM, Nando <d.n...@gmail.com> wrote:
> Aha! This connected a few dots for me. I so often wish the script based
> query construct was easier to use. Does the use of Clojure imply you wind up
> working with arrays of structs in ColdFusion?

Correct. Our data mapper traffics in arrays of structs in CFML and
sequences of (hash)maps in Clojure. Then we have a lightweight bean
adapter that wraps a struct to provide a getter/setter/OO interface.

In fact our bean adapter can also wrap an array of structs and provide
an iterator interface as well. Computed attributes are handled by
extension: write a CFC that extends our bean adapter and provide your
own set/get methods. The underlying bean is actually implemented with
onMissingMethod() so there's no real get/set methods.

> Running a bit off topic here but I saw Fogus tweet about Korma the other
> day. http://sqlkorma.com/ What do you think?

I don't like the entity-relationship mapping approach at all - not in
Hibernate / CFML, not in Korma - preferring a simpler data mapper.
Korma is built on top of the clojure.java.jdbc library which I
maintain, by the way.
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[John Berquist]

Sean,

If you can share, I would love to hear more about what you are doing here. Specifically, why are you wrapping the structs with a bean adapter to "provide a getter/setter/OO interface"? Why aren't you just work with the structs? You mentioned providing an iterator interface to an array of structs as well as making use of computed properties, are you doing anything else with your beans? I am very interested in your approach to data persistence, given your dislike of ORM, and would like to know more about your setup.

Thanks,

John

[Andrew Myers]

+1

:)

> --
> --
> FW/1 on RIAForge: http://fw1.riaforge.org/
>
> FW/1 on github: http://github.com/seancorfield/fw1
>
> FW/1 on Google Groups: http://groups.google.com/group/framework-one
>
> ---
> You received this message because you are subscribed to the Google Groups
> "framework-one" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to framework-on...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Sean Corfield]

On Tue, Feb 5, 2013 at 6:34 PM, John Berquist <jcber...@gmail.com> wrote:
> If you can share, I would love to hear more about what you are doing here.
> Specifically, why are you wrapping the structs with a bean adapter to
> "provide a getter/setter/OO interface"? Why aren't you just work with the
> structs?

The simplest answer:

user.getAge()

We have user.dateOfBirth as a data field. Age is computed. We don't
want to precompute all the "gettable" fields into a struct.

> You mentioned providing an iterator interface to an array of
> structs as well as making use of computed properties, are you doing anything
> else with your beans? I am very interested in your approach to data
> persistence, given your dislike of ORM, and would like to know more about
> your setup.

The bean really is a very thin OO veneer over the struct (or array of
structs). Setters just put data into a "dirty" struct. When you
.save() your bean, it has the changed data directly on hand and can
easily call Clojure to update the specific fields in the target
record.

In the bean's onMissingMethod(), we handle related objects on demand.
If we see getFoo() and there's no getFoo() method and no foo data
item, we look for fooId and if that exists we look in table foo for a
record with that id, and we cache it. If we see getFooIterator(), we
look in table foo for a FK barId that refers back to this bar object
and make an iterator and cache it. That's the extent of our "ORM".
Convention-based, on-demand... the bare minimum needed to make the
obvious stuff work without over-reaching. No performance pitfalls, no
complications.

Does that help explain things?

I'll be doing a talk at cf.Objective() called "ORM, noSQL, and
Vietnam" which will touch on some of this.

FWIW, our data mapper layer wraps both MySQL and MongoDB and our OO
wrapper works across both DBs. In fact we can have an object
MembershipEvent, backed by MongoDB, and call event.getUser() and
retrieve an object backed by MySQL. The data mapper knows which
"object" lives in which data store but the application doesn't need to
care - except insofar as it actually tries to do explicit SQL queries
or MongoDB aggregates (and I'm working on an abstraction to hide that
too).

[John Berquist]

Hi Sean,

Thanks for sharing, this is very informative. I have been fiddling with using a more sql-oriented non-ORM approach in CFML, using only structs and arrays of structs, so your posts in this thread (and the original) have been very timely for me. As you say, the fact that all computed fields and related "objects" have to be calculated and retrieved up front is an issue with going with a pure struct and array of structs approach, and has been a bit of a hang up for me. Your approach to this sounds very elegant (I particularly like your handling of getting related records) and I think you should share some code ;).

Along these lines, is there any way, or any plans to make a way, for DI/1 to handle transient beans by requesting them from a factory? So that if one requests a 'userBean', DI/1 could ask the factory for it before injecting any dependencies into it. This way, following your approach, one could make use of a generic bean by default (i.e. a user.cfc wouldn't have to exist in the beans folder) and then that bean could be extended as warranted, while still taking advantage of DI/1's dependency injection. Alternatively, could this logic be written into DI/1 itself?

Now I do see that you do describe your bean as "very thin"; do you handle all your CRUD business logic behind your Clojure data mapper, or do you put more into the beans? For instance, if you want to log something whenever a row is saved to a particular table in MySQL, do you take care of that in Clojure alongside of the UPDATE sql call, or do you inject some services into your beans and write a save method that calls your Clojure data mapper and then also calls a logging service?

Hopefully you will give your talk online at some point after cf.Objective() so I can hear it :).

-John

 

[Sean Corfield]

On Wed, Feb 6, 2013 at 4:52 PM, John Berquist <jcber...@gmail.com> wrote:
> bit of a hang up for me. Your approach to this sounds very elegant (I
> particularly like your handling of getting related records) and I think you
> should share some code ;).

The generic bean is very tightly tied to our Clojure data mapper and CFML / Clojure interop library but here's onMissingMethod() which automatically handles getFoo() and getBarIterator() (for a lookup in the foo table via this bean's fooId, and a lookup in the bar table for this bean's id as a foreign key) - it may not make any sense out of context but it might give you some ideas. It also supports getRelated("foo") and getIterator("bar") so that case sensitive table names can be used explicitly if the lowercase default doesn't work. I'll answer the other questions in a separate email.

public any function onMissingMethod( string missingMethodName, struct missingMethodArgs ) {

var stem = left( missingMethodName, 3 );

if ( stem == 'get' ) {

var root = right( missingMethodName, len( missingMethodName ) - 3 );

var n = len( root );

var itPost = "iterator";

var itLen = len( itPost );

if ( n >= itLen && right( root, itLen ) == itPost ) {

var startIndex = 1;

if ( n == itLen ) {

// getIterator("foo") or getIterator("foo",{ col = "asc|desc"}*)

if ( structKeyExists( missingMethodArgs, 1 ) ) {

var join = missingMethodArgs[ 1 ];

root = join & "iterator";

startIndex = 2;

} else {

throw "getIterator() requires the table name as an argument";

}

} else {

// getFooIterator() or getFooIterator({ col = "asc|desc"}*)

var join = lCase( left( root, n - itLen ) );

}

var orderInfo = { ordering = [] };

var key = join & "iterator";

if ( structKeyExists( missingMethodArgs, startIndex ) ) {

orderInfo = variables._collectOrderBy( missingMethodArgs, startIndex );

var key &= "|" & orderInfo.orderKey;

}

                // #3607: we can cache the _data_ but must create the

                // iterator afresh on every "get" operation:

if ( !structKeyExists( variables.relatedCache, key ) ) {

var fk = { };

fk[ variables.name & "Id" ] = this.getId();

variables.relatedCache[ key ] = variables.orm.findByKeys( join, fk, "query", orderInfo.ordering );

}

return variables.orm.createIterator( join, variables.relatedCache[ key ] );

}

var join = root;

var relatedConvention = true;

if ( root == "related" ) {

if ( structKeyExists( missingMethodArgs, 1 ) ) {

// override default all lowercase name

join = missingMethodArgs[ 1 ];

relatedConvention = false;

}

}

if ( !this._has( join ) ) {

// not a known property, check for possible related object

if ( this._has( join & "id" ) ) {

var key = join & "related";

if ( !structKeyExists( variables.relatedCache,key ) ) {

var relatedId = this._get( join & "id" );

if ( relatedConvention ) {

join = lCase( root );

}

variables.relatedCache[ key ] = variables.orm.create( join ).load( relatedId );

}

return variables.relatedCache[ key ];

}

}

return this._get( join );

} else if ( stem == 'has' ) {

var root = right( missingMethodName, len( missingMethodName ) - 3 );

return this._has( root );

} else if ( stem == 'set' ) {

var root = right( missingMethodName, len( missingMethodName ) - 3 );

if ( structKeyExists( missingMethodArgs, 1 ) ) {

return this._set( root, missingMethodArgs[ 1 ] );

} else {

throw "#missingMethodName#() called without an argument";

}

} else if ( stem == "nil" ) {

var root = right( missingMethodName, len( missingMethodName ) - 3 );

return this.nil( root );

} else {

var message = "no such method (" & missingMethodName & ") in " & getMetadata(this).name & "; [" & structKeyList(this) & "]";

throw "#message#";

[Sean Corfield]

On Wed, Feb 6, 2013 at 4:52 PM, John Berquist <jcber...@gmail.com> wrote:

Along these lines, is there any way, or any plans to make a way, for DI/1 to handle transient beans by requesting them from a factory? So that if one requests a 'userBean', DI/1 could ask the factory for it before injecting any dependencies into it. This way, following your approach, one could make use of a generic bean by default (i.e. a user.cfc wouldn't have to exist in the beans folder) and then that bean could be extended as warranted, while still taking advantage of DI/1's dependency injection. Alternatively, could this logic be written into DI/1 itself?

A very interesting idea. Could you open an issue on the DI/1 github site with some use cases? 

Now I do see that you do describe your bean as "very thin"; do you handle all your CRUD business logic behind your Clojure data mapper, or do you put more into the beans? For instance, if you want to log something whenever a row is saved to a particular table in MySQL, do you take care of that in Clojure alongside of the UPDATE sql call, or do you inject some services into your beans and write a save method that calls your Clojure data mapper and then also calls a logging service?

The need hasn't arisen to log changes at that level. We do have a logging service that we make explicit calls to throughout our application but only one object needs to have a full audit history of its changes so it takes care of that explicitly.

Our beans are "very thin" only insofar as the amount of "wrapping" around the raw data mapper, e.g., bean.save() pretty much just delegates to the Clojure code:

public any function save( boolean reload = false ) {

if ( !variables.loaded ) return;

if ( structKeyExists( variables.data, variables._pk() ) && !structKeyExists( variables.dirty, variables._pk() ) ) {

variables.dirty[ variables._pk() ] = variables.data[ variables._pk() ];

}

var id = variables.clj.worldsingles.data.save_row( variables.name, variables.dirty, variables._keyGenPolicy() );

if ( reload ) {

this.load( id );

} else {

for ( var key in variables.dirty ) {

variables.data[ key ] = variables.dirty[ key ];

}

variables.data[ variables._pk() ] = id;

variables._cleanData();

}

return this;

}

The variables.clj object is our hook into Clojure and our entire Clojure code base is available as sub-objects of that (and, again, we use onMissingMethod() to map function calls to lookups of Clojure Var names and then call .invoke() on the returned IFn implementations).

Hopefully you will give your talk online at some point after cf.Objective() so I can hear it :).

Yup, I expect I'll get around to giving all three of my cf.Objective() talks via the online meetup at some future date...

[Sean Corfield]

On Wed, Feb 6, 2013 at 7:10 PM, Sean Corfield <seanco...@gmail.com> wrote:

The variables.clj object is our hook into Clojure and our entire Clojure code base is available as sub-objects of that (and, again, we use onMissingMethod() to map function calls to lookups of Clojure Var names and then call .invoke() on the returned IFn implementations).

I thought I'd follow up on this to show some CFML / Clojure interop...

This is in the init() of our ORM service CFC:

        // executeCount assumes SELECT COUNT(*) FROM ...

        var core = variables.clj.clojure.core;

        this.executeCount = core.comp(

            core._first(),

            core._vals(),

            core._first()

        );

Our Clojure interop layer lets you call functions directly, via onMissingMethod() as I mentioned above, but it also lets you get a reference to the function itself, instead of calling it immediately, by calling _<functionname>() instead - or calling _("functionname") but I prefer the former syntax. So the above code reaches into the Clojure core library - essentially the language itself - and executes the equivalent of this Clojure expression:

    (comp first vals first)

That composes three functions to make a new function. That anonymous (Clojure) function is then stored as a public variable in the ORM service CFC directly. It's used like this:

public boolean function isHighRisk( string country ) {

var highRiskCountry = variables.orm.execute(

"SELECT COUNT(*)

         FROM countryRisk

WHERE ISO = ?", 

[ country ],

variables.orm.executeCount

);

return highRiskCountry > 0;

}

orm.execute() simply delegates to Clojure:

public any function execute() { // sql-string, params = [ ], transform = #(doall (map cfml/to-struct %)) 

return variables.clj.worldsingles.data.execute( argumentCollection = arguments );

}

Here we pass the CFML arguments directly into Clojure(!) so it is equivalent to this Clojure code:

    (worldsingles.data/execute "SELECT COUNT(*) FROM countryRisk WHERE ISO = ?" [country] variables.orm.executeCount)

For that function, the last (optional) argument is a function which is applied to the resultset before it is returned. Behind the scenes, a JDBC PreparedStatement is created with that SQL and the parameters, the vector containing country, and executed and then executeCount is applied to the resultset - a vector of maps (which will contain just one element, a map with a key of "COUNT(*)" and the appropriate count as its value).

So, in Clojure, we have the equivalent of:

    ((comp first vals first) [{"COUNT(*)" 1}])

and that is equivalent to:

    (first (vals (first [{"COUNT(*)" 1}])))

which evaluates to:

    (first (vals {"COUNT(*)" 1}))

    (first [1])

    1

and so execute() returns 1. Without the function executeCount being passed, the execute() function would return a CFML array containing CFML struct: { "COUNT(*)" = 1 } - we convert between CFML arrays and Clojure vectors, and between CFML structs and Clojure maps, all automatically in our interop layer.

Note that ((comp a b c) x) ;; is equivalent to (a (b (c x))) or in CFML script: a( b( c( x ) ) );

We call a number of Clojure core functions directly from CFML to do different things, such as map functions over sequences of data (in the Map/Reduce sense, not in the hash map sense), build lists and (hash) maps and so on.

-- 

[John Berquist]

Thanks Sean! This is all very helpful, particularly the onMissingMethod() code. Thanks also for sharing some of your Clojure integration, what you are doing looks very cool. Reading the first post I was wondering what all of the _functionName() calls were :).  I have probably heard or read this before, but I am not very familiar with Clojure: are Railo and Clojure running on the same JVM? I don't know much about such things, but if that is right, how difficult/tricky was it to get that set up? Do you see yourself sticking with CFML for the front end, or are you eventually going to move away from it there as well?

I opened an issue with a feature request for DI/1 with the behavior I would like to see added. I think I was over thinking it a bit last night, but I have my actual use case presented there so you can consider it for yourself.

-John

[Sean Corfield]

On Thu, Feb 7, 2013 at 7:06 AM, John Berquist <jcber...@gmail.com> wrote:

I have probably heard or read this before, but I am not very familiar with Clojure: are Railo and Clojure running on the same JVM?

Yup, we use Clojure for heavy lifting instead of Java (I can't stand Java - it's so verbose!) so we have the Clojure libraries on the Tomcat classpath, where we run Railo, and I use https://github.com/seancorfield/cfmljure to glue together so that we can call directly into Clojure from CFML and even pass CFML data structures back and forth. It's a very powerful combination.

 

I don't know much about such things, but if that is right, how difficult/tricky was it to get that set up?

The basics are easy - just put clojure-1.4.0.jar on your classpath so you can do createObject( "java", "clojure.lang.RT" ) - the Clojure runtime, then you have clojure.core and a few other libraries and, via the onMissingMethod() magic of cfmljure, you can call Clojure code from CFML.

The full setup requires that your Clojure source code tree is also on the classpath - Clojure locates files via the classpath rather than by absolute paths and then compiles source to JVM bytecode on the fly on first reference, just like CFML. We actually modify the catalina.properties file to extend the common class loader path, then copy library JARs into WEB-INF/lib/ and have our Clojure source outside the webroot (in a project tree managed by Leiningen).

Once the setup is done once, you never need to touch it again, except for copying in the JAR for any new library dependency you need. We have about 50 JARs pulled in automatically - Clojure often relies on Java libraries in the same way CFML does... and ColdFusion / Railo already have a bunch of Java libraries as JARs in the lib/ folder.

 

Do you see yourself sticking with CFML for the front end, or are you eventually going to move away from it there as well?

Our interim goal is CFML for View-Controller and Clojure for Model. Once we've migrated off ColdBox onto FW/1 (a long ways out), we may then consider moving to FW/1 for Clojure and switch the whole stack to Clojure but that's years away...

 

I opened an issue with a feature request for DI/1 with the behavior I would like to see added. I think I was over thinking it a bit last night, but I have my actual use case presented there so you can consider it for yourself.

Thank you!