NoSQL with ColdFusion, Bean+Service+DAO & OOP or good old Array/Struct & Procedural?

[Henry]

How do you architect the CF backend model w/ NoSQL that are simple, flexible, efficient and clean?

Since NoSQL doc has no fixed schema like SQL row, it doesn't really fit well with Objects which are rather static. Therefore the typical Bean+DAO+Service OOP architecture doesn't seem to fit well.

I'm thinking of using plain old Struct's, but then I cannot add behavior onto it and it's going to make the whole project very procedural, which may not be a bad thing?

However, if I just use plain old struct, the DB implementations is leaked everywhere including the View layer...

Or... shall I translate the array's into CF's Query object for the View layer?

Comment? Idea? Suggestion?

Thanks!

p.s. also asked here: http://stackoverflow.com/questions/4622121/nosql-with-coldfusion-beanservicedao-oop-or-good-old-array-struct-procedur

[Peter Bell]

I'd look into onMissingMethod() so you can call arbitrary attributes. I certainly wouldn't get rid of objects. You're going to have to do a bit of magic to handle composed objects - probably by telling your code about it, so if you try to getName(), it'll return a String, but getComments() will return an iterator of Comment objects. A little metadata for known relationships should suffice, and if you have arbitrary nested objects, a generic iterator returning instances of a generic object which has generic getters. 

Best Wishes,

Peter

--
You received this message because you are subscribed to the Google Groups "CFCDev" group.
To post to this group, send email to cfc...@googlegroups.com.
To unsubscribe from this group, send email to cfcdev+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cfcdev?hl=en.

[Barney Boisvert]

I don't think that a non-sql database and a strongly typed object
model are incompatible at all. Consider a blog (yeah, I know), you
might have blog, author, entry, and comment entities, along with blog,
author, and entry collections in your database. Comments are stored
as an array of subdocuments within the appropriate entry document.

Regardless of the implementation, your in-memory domain model and your
durable persistence are totally divorced from each other. Or at least
they should be. There's obviously glue code that is implementation
specific for doing persistence operations, but that should be
encapsulated. So aside from that glue code, you shouldn't have to
change much architecturally.

Obviously if you're looking at some of the large-scale benefits of a
non-sql database (e.g., Couch's autosharding) that's gonig to
influence your application design. But the "non-sql" aspect isn't
doing the influencing - you'd need the same design considerations if
you were using a RDBMS that did autosharding.

cheers,
barneyb

> --
> You received this message because you are subscribed to the Google Groups
> "CFCDev" group.
> To post to this group, send email to cfc...@googlegroups.com.
> To unsubscribe from this group, send email to
> cfcdev+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/cfcdev?hl=en.
>

--
Barney Boisvert
bboi...@gmail.com
http://www.barneyb.com/

[Peter Bell]

The one difference is that many NoSQL stores don't have a fixed schema, so you have the option of having arbitrary properties. That's what I'd handle with oMM()

[Henry Ho]

Peter,

I was doing query with NoSQL yesterday. The result set is array of structs.  Then it reminded me of the IBO pattern that you introduced to me. :)

Do you think this is a good use case to wrap the array of structs into an IBO?  Maybe that's a better approach to.. translating an array (MongoDB query result set array) into a query object?

Henry

[Henry Ho]

Barney B,

Ya, the thing is, docs in noSql are quite dynamic (no fixed schema) and should be easy to change.

Maybe you're right.  When it comes to change, with the Bean/DAO/Service way, I had to change the DAO and Bean, but at least the view layer doesn't have to change, much.

With plain old struct, I can minimize the changes of Bean/DAO, and just update the Service layer and 'leak' the change(s) into the View layer.

Pick my poison I guess?

Henry

[Peter Bell]

Well, I certainly wouldn't use a query object. I'm not a fan as there is no possibility of encapsulation of any possibly business logic.

These days, I don't like the IBO too much either. I always knew it was violating the Single Responsibility Principle by being both an object and an iterator, but it worked pragmatically so I used it. These days, I usually create an Iterator which returns an object, but instead of creating an array of object and wrapping them in an iterator I always return the same object, but loaded with different state. 

I would definitely recommend objects for all the usual reasons (unless you believe you're always going to be doing pure dumping of data to the screen with no business logic at all).

Best Wishes,

Peter

[Barney Boisvert]

> Ya, the thing is, docs in noSql are quite dynamic (no fixed schema) and
> should be easy to change.

Yes, that's one of the much-stated benefits - and it's a very
compelling one - but if you change the structure of a document you
still have to change code for it to DO anything. I.e. your code is
still dependant on the data in the DB, you just are absolved from a)
forcing every instance of a collection to have identical fields, and
b) formalizing data structure changes with DDL.

For example, say I want to add a URL slug to my contrived blog app.
Yes, I can just start throwing 'slug' attributes into my entries in my
database, but they don't do anything until I expose that data in the
application (probably editing, but certainly storing at creat time and
use for post selection).

Short answer, regardless of your storage mechanism, if your data
changes your application must change.

Of the two points I listed above, I personally thing than 'a' is of
FAR greater benefit than 'b'. It's ridiculously powerful to just
throw extra stuff on some entities in your collection without having
to consider the rest of the collection. You, of course, have to wrap
it with some business logic for identifying which entities are
"special" in that way, but being able to just toss stuff in there is
the big win of schema-less data stores.

cheers,
barneyb

On Fri, Jan 7, 2011 at 11:02 AM, Henry Ho <henry...@gmail.com> wrote:
> Barney B,
> Ya, the thing is, docs in noSql are quite dynamic (no fixed schema) and
> should be easy to change.
> Maybe you're right.  When it comes to change, with the Bean/DAO/Service way,
> I had to change the DAO and Bean, but at least the view layer doesn't have
> to change, much.
> With plain old struct, I can minimize the changes of Bean/DAO, and just
> update the Service layer and 'leak' the change(s) into the View layer.
> Pick my poison I guess?
>
>
> Henry
>

--

[Henry Ho]

So it'll be something like...

iterator.next() that returns the actual object, then calling iterator.next() uses the same instance, but overwrite the memento of the 2nd struct in the array?  Interesting... 

Henry

[Henry Ho]

Yes, that's one of the much-stated benefits - and it's a very
compelling one - but if you change the structure of a document you
still have to change code for it to DO anything.  

Yes, I get that, but with the DAO/Service/Bean approach, it seems like I have more layers needed to be changed.  I'm just exploring if there're any other lighter/more flexible approach to this.

Since I'm using MongoDB, the results from the driver (i.e. http://mongocfc.riaforge.org/) is already in struct or array.  I'm just wondering if I can just use them and accelerate the development time, while keep code as clean as possible.

Not that I'm really in love with dealing only with data (struct/array), just thinking out loud, and what could I have done better.

Thanks,

Henry

[Peter Bell]

On Jan 7, 2011, at 2:43 PM, Henry Ho wrote:

Yes, I get that, but with the DAO/Service/Bean approach, it seems like I have more layers needed to be changed.  I'm just exploring if there're any other lighter/more flexible approach to this.

I don't think there is anything wrong with having a DAO, Service and Bean, but there shouldn't be any real duplication of metadata between them. There should be a single authoritative source of any piece of data in your app. So you need to pick one place to tell your system what properties a given object has. 

IMO, if you need to edit more than one thing when you add a property to an object, you're doing it wrong. I always have a single place with metadata describing my objects (either putting it into the object and using instantiation and reflection or having a metadata bean that I load into all of the objects relating to a given business object) and use that to control data access, the object itself and anything in the service class that needs to know about the business object. Where I can, I also use that representation to generate my DB schema if I have one (or to generate migrations) so I'm keeping things nice and DRY.

Best Wishes,

Peter

[Steve Bryant]

Peter,

Would you mind expanding on that a bit? How does the use of a query
prevent encapsulation? I realize that the data is read-only, but that
doesn't mean that the code that produced it wasn't well-encapsulated.

Or am I missing something fundamental?

Thanks,

Steve

[Barney Boisvert]

He means you can't encapsulate business logic within the entities
you're iterating over. I.e., a query row is completely "dumb" - the
worst case scenario for an anemic domain model.

As a very simple example, if you have a recordset with id, name, and
dateOfBirth columns, you have to do age calculations in every
view/controller that needs an age value. But if you have an array of
objects, you can just expose a 'getAge' method that encapsulates the
logic, thereby reducing the duplication all over your app.

<cfloop query="getPeople">
...
<td>#dateDiff('yyyy', getPeople.dateOfBirth, now())#</td>
...
</cfloop>

vs.

<cfloop array="#people#" index="person">
...
<td>#person.getAge()#</td>
...
</cfloop>

I believe the encapsulation you were referring to (in the code
producing the recordset) is the implementation of PersonDao's
findPeople(...) method, right? No way to tell if that's SQL backed,
SOAP backed, etc., which is good, but it's the less interesting
encapsulation, because it's not business logic, it's just data access
"stuff". The encapsulation of business logic is in your domain model,
not your data access code.

cheers,
barneyb

[Peter Bell]

Hey Steve,

Long time no chat! Hope you're doing well?

OK, trivial example, I get firstName and lastName back and I want to display fullName. If I don't have an object where I can put a getFullName() method, I have to either iterate over the query in a service class to preprocess or duplicate the #firstName# #lastName# code in every view which is fine until I want to add middleInitial to everywhere I display the full name and then becomes a PiTA in terms of maintenance.

Of course, if all we ever did was to display a fullName, this would be overkill, but I find complexity usually grows. For example, I have a location object in a current app. You enter an address, but you can call location.lat and location.lng which return the lat and lng. Inside the location bean you have:

def lat
if(!@lat)
latlng = GeoKit::Geocoders::GoogleGeocoder.geocode("#{self.full_address}")
self.lat = latlng.lat
self.lng = latlng.lng
end
latlng.lat
end

It's ruby code, but you get the idea. I'm populating the latlng from a Google geocoding lookup. The alternative approach of pre-processing your query to populate all of the lats and longs works, but the more calculated properties you have, the more complex and difficult to maintain those pre-processing routines become.

Such preprocessing routines also violate (to me) the single responsibility principle. Sure, they are responsible for "preprocessing the query to generate calculated values", but that's a big responsibility that could include concatenating fields, pulling from third party web services and who knows what else to create the fully populated queries with all of the calculated properties.

I just default to objects with smart getters and seldom find myself regretting it.

Also much easier to unit test the calculations.

Best Wishes,
Peter

[Henry Ho]

I have another idea, see what you guys think.

Since the NoSQL DB driver I'm using is returning me a Struct for a doc (most of them do), could I have just write a proxy that wraps the struct, and add behaviors on it? instead of decompose the Struct into properties of a bean?

The pro is that I don't really need a DAO per Bean, but I'll need to write some smart getter/setter's or implement a smart onMissingMethod.  And when I need to attach some special property for some particular instances, I can just do so and attach to the struct.

What do you think?

Thx,

Henry

[Steve Bryant]

Peter,

Indeed! I'm excellent. Hope you are well also.

I have a "fieldlist" argument on my service methods. So, my
"getEmployees" method returns a query with the fields requested. It
doesn't matter that some of those fields are created by SQL and others
by post-processing. At which point I am only doing post-processing for
the pages that need it.

Internally, I can have those calculations done by other methods for unit
testing.

For me, this has the advantages of dealing with easy queries and of
being able to have a rule for "no method calls in the view" which helps
to avoid inadvertent calls to looping code from within an output loop.

I guess I still don't see how this violates encapsulation (I'm not even
sure that the issue mentioned is truly one of encapsulation). I am a
missing something?

Thanks,

Steve

[Peter Bell]

On Jan 7, 2011, at 4:46 PM, Steve Bryant wrote:

> Peter,

> Indeed! I'm excellent. Hope you are well also.

A little busy, but doing very well!

> I have a "fieldlist" argument on my service methods. So, my "getEmployees" method returns a query with the fields requested. It doesn't matter that some of those fields are created by SQL and others by post-processing. At which point I am only doing post-processing for the pages that need it.
> Internally, I can have those calculations done by other methods for unit testing.

So I'm guessing you have something like:

userQuery = UserService.getAllUsers("firstName, lastName, age, fullName, homeLatitude, homeLongitude")

I guess if you have some logic which determines generically which fields are from the db and which are calculated, you could write well encapsulated code which would first construct a base query using SQL and then "for each calculated field" would call:

newQuery = addAgeToQuery(currentQuery)

So then you'd call addFullNameToQuery(query) then addHomeLatitudeToQuery(query), then addHomeLongitudeToQuery(query)

I can see how that could work. In effect you're adding your smart getters to your service class instead of your business object and instead of them taking no parameter and returning a value, they take a query and return a query augmented with the calculated column.

There's no reason this wouldn't work. However, it's not how everyone else in the world writes apps. The vast majority of people writing complex apps that aren't using a functional style (F#, Scala, Clojure, Erlang, Haskell, etc) are using an OO approach. With an OO approach, you put those smart getters as instance methods on a business object.

Most other devs, most tooling, most design patterns and most other elements of the craft of software development are using the standard OO approach, and with the exception of performance issues that are becoming fairly unimportant, I don't think the "put all the methods into the service class" is substantially better than the OO approach (I'd suggest that it is marginally worse, but not unusably so).

It seems to me that by a combination of ColdFusions historically bad object creation penalty and the way DataMapper and your coding style has evolved, you're in a risk of painting yourself into a local optima that is not a global optima. I would imagine that if you tried to write an app in a more OO style it would take you longer than to create the app the way you do now (not least due to the fact that you have good familiarity and have built a good tool chain around your approach). The problem is that with a substantial part of the global programming community going in a different direction, over time the tooling supporting the OO approach is going to supersede what you have and you're going to be increasingly less able to compete and/or work with other devs used to the "more popular" approach.

> For me, this has the advantages of dealing with easy queries

I don't see queries as easier. I accept they may be more familiar to some CF devs, but I'd argue once you're familiar with them, objects are as easy to work with as queries - and are more flexible.

> and of being able to have a rule for "no method calls in the view" which helps to avoid inadvertent calls to looping code from within an output loop.

That's pretty much the opposite of how any OO app works. ALL of them are based on calls in the view to both business objects and view helpers. If you had no method calls in the view in Rails or Grails, you wouldn't have an application at all.

Personally, I'd reconsider that rule.

Best Wishes,
Peter

[Barney Boisvert]

Rather than just reconsidering the rule, you might simply replace it
with "no multi-term expressions in the view". Not a viable goal in
all cases, but pretty darn close. Views shouldn't have logic apart
from that directly concerned with the view structure (looping rows in
a table, conditionally displaying an action button, etc.). All the
dynamic values and conditional expressions should be single terms
(often - usually? - a method call):

#person.getAge()#
not
#dateDiff('yyyy', dateOfBirth, now())#

<cfif person.isMinor()>
not
<cfif person.getAge() GT AGE_OF_MAJORITY>

cheers,
barneyb

[Peter Bell]

I *like* that rule :)

[Steve Bryant]

Peter,

> So I'm guessing you have something like:
>
> userQuery = UserService.getAllUsers("firstName, lastName, age, fullName, homeLatitude, homeLongitude")

Pretty much:

qUsers =
Application.Users.getUsers(hasEmailAddress=true,fieldlist="firstName,

lastName, age, fullName, homeLatitude, homeLongitude")>

(for example)

> I guess if you have some logic which determines generically which fields are from the db and which are calculated, you could write well encapsulated code which would first construct a base query using SQL and then "for each calculated field" would call:
>
> newQuery = addAgeToQuery(currentQuery)
>
> So then you'd call addFullNameToQuery(query) then addHomeLatitudeToQuery(query), then addHomeLongitudeToQuery(query)
>
> I can see how that could work. In effect you're adding your smart getters to your service class instead of your business object and instead of them taking no parameter and returning a value, they take a query and return a query augmented with the calculated column.

That would work. I actually have empty fields in my query for the
calculated fields and then if any of them exist, I loop over the query
(once) in the method and do something like this for each row.
<cfset
QuerySetCell(qUsers,"fullName",makeFullName(firstName,lastName),CurrentRow)>
I may have to ensure that firstName and lastName are in the original
query if the fullname column is requested, but that is trivial. This
makes the makeFullName internal method a bit easier to unit test.

> There's no reason this wouldn't work. However, it's not how everyone else in the world writes apps. The vast majority of people writing complex apps that aren't using a functional style (F#, Scala, Clojure, Erlang, Haskell, etc) are using an OO approach. With an OO approach, you put those smart getters as instance methods on a business object.
>
> Most other devs, most tooling, most design patterns and most other elements of the craft of software development are using the standard OO approach, and with the exception of performance issues that are becoming fairly unimportant, I don't think the "put all the methods into the service class" is substantially better than the OO approach (I'd suggest that it is marginally worse, but not unusably so).

I think this is a separate issue from encapsulation. I do see some
benefit in doing things the popular way, but not enough to be
determinative. Certainly, if I were using another language then that
would affect my decision heuristic. In JavaScript, for example, I do use
Objects. That is natural to how JavaScript works.

Within the ColdFusion universe, I don't particularly think OO is
dominant - despite the appearance among those of us on lists like this
or reading and writing ColdFusion blogs. Again, not that I find that
determinative.

> It seems to me that by a combination of ColdFusions historically bad object creation penalty and the way DataMapper and your coding style has evolved, you're in a risk of painting yourself into a local optima that is not a global optima. I would imagine that if you tried to write an app in a more OO style it would take you longer than to create the app the way you do now (not least due to the fact that you have good familiarity and have built a good tool chain around your approach). The problem is that with a substantial part of the global programming community going in a different direction, over time the tooling supporting the OO approach is going to supersede what you have and you're going to be increasingly less able to compete and/or work with other devs used to the "more popular" approach.

Only if I don't also make myself cognizant of other "more popular"
approaches. ColdFusion's bad object creation penalty is non-trivial here
as well. Queries are stinking fast by comparison.

> I don't see queries as easier. I accept they may be more familiar to
> some CF devs, but I'd argue once you're familiar with them, objects
> are as easy to work with as queries - and are more flexible.

This is a whole other topic. I'll try to write a blog entry on it
sometime. For now, we'll have to agree to disagree. Still, separate from
the encapsulation argument.

> That's pretty much the opposite of how any OO app works. ALL of them
> are based on calls in the view to both business objects and view
> helpers. If you had no method calls in the view in Rails or Grails,
> you wouldn't have an application at all.

Well, I'm not doing OO or Rails. If I were using Rails, I would be doing
OO and I would not have that rule.

Thanks,

Steve

[Steve Bryant]

> Rather than just reconsidering the rule, you might simply replace it
> with "no multi-term expressions in the view". Not a viable goal in
> all cases, but pretty darn close. Views shouldn't have logic apart
> from that directly concerned with the view structure (looping rows in
> a table, conditionally displaying an action button, etc.). All the
> dynamic values and conditional expressions should be single terms
> (often - usually? - a method call):
>
> #person.getAge()#
> not
> #dateDiff('yyyy', dateOfBirth, now())#
>
> <cfif person.isMinor()>
> not
> <cfif person.getAge() GT AGE_OF_MAJORITY>
>
> cheers,
> barneyb

For me, I would probably pass AgeOfMajority into my service component
and have it return an "isMinor" column. If that wasn't a possibility,
then I would have my page controller add that column to the query.

I would also have an "age" column in the query returned from the service
(assuming it was in the fieldlist).

I like a clean view with no method calls.

Steve

[Barney Boisvert]

By front-computing all those values, you're necessarily doing a LOT of
extra work to build up data that isn't needed universally. A dataDiff
for age is simple, but what if it's numberOfGrandchildren (which is
guaranteed to require database access to answer). Using a method for
lazy loading the data can be a HUGE performance boon, as well as
moving a huge amount of entity-specific logic into the entity (rather
than in external components divorced from the entity itself).

But like you said to Peter, I think we have to agree to disagree on this. :)

cheers,
barneyb

> --
> You received this message because you are subscribed to the Google Groups
> "CFCDev" group.
> To post to this group, send email to cfc...@googlegroups.com.
> To unsubscribe from this group, send email to
> cfcdev+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/cfcdev?hl=en.
>
>

--

[Steve Bryant]

Barney,

That's why I pass a fieldlist argument in. If "numberOfGrandchildren"
isn't in the fieldlist then I don't calculate it.

Does that not cover that case?

Steve

[Bob Silverberg]

That is what I do in ValidateThis when validating a struct rather than
an object. You'd need to have a way of easily adding behaviours to the
wrapper object. Perhaps a base object that just wraps, and if you need
extra behaviours you could extend the base wrapper with a concrete
wrapper for that object?

Bob

--
Bob Silverberg
www.silverwareconsulting.com