ORMs and clean architecture?

[Daniel Karp]

This is related to my other thread about persistence as a detail and testing, and is particularly aimed at any fellow readers/watchers who use PHP.

I started writing a small application deferring decisions about persistence for a while--when it came time to implement the database I found that it was very tempting to implement the persistence using an ORM (object-relational mapper), because I thought it would allow me to store my entities with not much change in the way they were implemented.

The issue is this: most ORMs, in PHP anyway, require your entities to extend some base class of the ORM. Of course, that is a definite no-no for clean architecture--the Entities should know nothing about the persistence layer. Theoretically, I could put those ORM-aware "entities" entirely in the Database implementation, and translate them into my application entities in that layer, but having two sets of entities seems kind of silly.

Doctrine ORM is a bit different; it does NOT require your entities to derive from a common parent at all, so I thought, "great, I can use these in a clean way". I implemented the application with Doctrine, but now I think I was a bit fooled by not understanding the details of Doctrine: although the entities do not extend from a common parent, the objects returned by the entity manager, when the objects are retrieved from the database, are NOT purely instances of your entity objects; rather, they are instances of Proxy classes, created by doctrine, that supposedly behave just like your entities, but override some of the getters to allow for lazy-loading of relationships.

To me, this seems almost as bad as, if not worse than, having the entities derive from some class in the ORM. If the entity manager returns objects that are not really instances of my classes, how can I trust the tests that DON'T use doctrine for the persistence? 

Is using an ORM just incompatible with clean architecture? Am I worrying too much about the use of proxy classes by Doctrine? 

[Dave Marshall]

Hi Daniel

I'm no expert, I'm probably at a similar stage to you, but seeing as I work with PHP a lot, I figured I better chime in, I'm not sure how many PHP users will be lurking on this list :/

With regards to doctrine, I think you are worrying about the proxies a bit too much. As far as I know, it's quite a common technique used by Data Mapper implementations, I'm sure Hibernate and such will do similar things. 

Doctrine is a well built ORM, with a good development team, a good test suite and a plethora of people testing it day to day (in or out of production). I trust it well enough, but I understand your misgivings, all abstractions are leaky.

[John Stoneham]

I find ORMs are just fine for clean architecture, but they need to be a detail. They've got to be hidden behind your persistence layer interface or you're going to hurt eventually. But as a way to implement that interface, they may be great.

That means: it's fine if the entities have to extend some ORM class, but you'll have two trees of entities. Or, if your ORM is sufficiently magical, you may be able to use your business entities and map them in, though I wouldn't recommend it because you often want the persistence implementation and the business entities to vary independently. Your persistence implementation will change due to changes in your business objects, but the reverse should never be true. If your persistence implementation returns magical proxies, those proxies should be there to enable some well-specified feature of your persistence layer, so any behavior you're worried about ought to be covered by integration tests on the persistence implementation.

Lazy-loading should be an optimization to help you have to manage the object graph less explicitly for use cases, but it can be a sign you haven't thought carefully about the data needs of some particular use case when invoking it. It generally adds round-trips to the database, which can harm performance, and opens you to invisible n+1 query problems or memory exhaustion on big relations, so you're trading those concerns for simpler management of what data to load before starting work.

When I look at designing integrations with ORM, I often think, "what would have to change if I decided to use raw SQL, noSQL, or REST persistence instead?" The ideal answer is, "only the implementations of the interface of my persistence layer, and nothing in any other module except maybe main, and I'd know it would work when I was finished because all the integration tests passed."

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discu...@googlegroups.com.
To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.
 
 

--
John Stoneham
ly...@lyrically.net

[Daniel Karp]

Thank you very much for your thoughtful reply!

I did try to use my business entities as my ORM entities, and it seemed fine to begin with, but gradually became a mess. The biggest symptom that things weren't right was when I found that I could do pass some tests with the ORM repository implementation  but not with my initial in-memory database, suggesting that I was starting to lean on the ORM to handle business rules.

So I think it is back to the drawing board for object persistence, for me.

On Sunday, June 16, 2013 9:41:03 PM UTC-4, John Stoneham wrote:

I find ORMs are just fine for clean architecture, but they need to be a detail. They've got to be hidden behind your persistence layer interface or you're going to hurt eventually. But as a way to implement that interface, they may be great.

That means: it's fine if the entities have to extend some ORM class, but you'll have two trees of entities. Or, if your ORM is sufficiently magical, you may be able to use your business entities and map them in, though I wouldn't recommend it because you often want the persistence implementation and the business entities to vary independently. Your persistence implementation will change due to changes in your business objects, but the reverse should never be true. If your persistence implementation returns magical proxies, those proxies should be there to enable some well-specified feature of your persistence layer, so any behavior you're worried about ought to be covered by integration tests on the persistence implementation.

Lazy-loading should be an optimization to help you have to manage the object graph less explicitly for use cases, but it can be a sign you haven't thought carefully about the data needs of some particular use case when invoking it. It generally adds round-trips to the database, which can harm performance, and opens you to invisible n+1 query problems or memory exhaustion on big relations, so you're trading those concerns for simpler management of what data to load before starting work.

When I look at designing integrations with ORM, I often think, "what would have to change if I decided to use raw SQL, noSQL, or REST persistence instead?" The ideal answer is, "only the implementations of the interface of my persistence layer, and nothing in any other module except maybe main, and I'd know it would work when I was finished because all the integration tests passed."

On Sun, Jun 16, 2013 at 10:26 AM, Daniel Karp <danie...@gmail.com> wrote:

This is related to my other thread about persistence as a detail and testing, and is particularly aimed at any fellow readers/watchers who use PHP.

I started writing a small application deferring decisions about persistence for a while--when it came time to implement the database I found that it was very tempting to implement the persistence using an ORM (object-relational mapper), because I thought it would allow me to store my entities with not much change in the way they were implemented.

The issue is this: most ORMs, in PHP anyway, require your entities to extend some base class of the ORM. Of course, that is a definite no-no for clean architecture--the Entities should know nothing about the persistence layer. Theoretically, I could put those ORM-aware "entities" entirely in the Database implementation, and translate them into my application entities in that layer, but having two sets of entities seems kind of silly.

Doctrine ORM is a bit different; it does NOT require your entities to derive from a common parent at all, so I thought, "great, I can use these in a clean way". I implemented the application with Doctrine, but now I think I was a bit fooled by not understanding the details of Doctrine: although the entities do not extend from a common parent, the objects returned by the entity manager, when the objects are retrieved from the database, are NOT purely instances of your entity objects; rather, they are instances of Proxy classes, created by doctrine, that supposedly behave just like your entities, but override some of the getters to allow for lazy-loading of relationships.

To me, this seems almost as bad as, if not worse than, having the entities derive from some class in the ORM. If the entity manager returns objects that are not really instances of my classes, how can I trust the tests that DON'T use doctrine for the persistence? 

Is using an ORM just incompatible with clean architecture? Am I worrying too much about the use of proxy classes by Doctrine? 

--

John Stoneham
ly...@lyrically.net

[daniphp]

Hello,

ORM are just Structure Generators. They take the schema and generate the Pseudo-Models(Structures). It doesn't matter if they use Active Record or Data Mapper as a pattern. In the end your goal is the same: to have the Business Objects separated from the Database implementation.  I think the biggest problem for as as developers is to start forgetting the database structure and concentrate on Entities. Even if they are similar as structure in 90% of cases for simple web applications they will get bigger and the separation will happen at some point.

Daniel

[Andreas Schaefer]

using an ORM you have to map the "PlugIn entities" to the business logic entities, either manually or by the help of automation.
persistance done with web services (if restful or not) is a good example for an alternative. there you'd have to map the dynamic json (or whatever plain data structure) to the business logic entities.
so you see .. some devs call it a luxury to have statically typed objects to map against ;)
others (e.g. using document / key value stores) prefer the absence of a schema and the freedom it gives you.

either way, it's plain simple: the business logic entities and interactors are the most abstract part of your system. they don't have dependencies to components with a lower level of abstraction (e.g. ORMs that only Plug in to your system)

[Uncle Bob]

Using an ORM is not incompatible with Clean Architecture.  However, the ORM belongs below the line separating the entities from persistence.  In other words, the ORM is part of the persistence component.  

Yes, this means that there will be some duplication between your ORM data structures and your business entities.  However, those ORM data structures do not have business methods, and your business entities do.  To get the dependencies right, it is possible to create business entities that have abstract getters that are implemented by the ORM data structures below the line.  This eliminates the duplication, but at the expense of the tight coupling of an inheritance relationship.

[Zamith]

What about validations provided by the ORM, would you duplicate them as well? Would you treat them as business rules and simply not use them on the persistence component?

[James Green]

Aren't you mentally mapping ORM to persistence by saying that? There are similar mappers (omitting the world "Relational") consuming and producing entities for public consumption - surely this is where validation takes place? By the time you have hit the persistence component your data should be valid.

--

[Luis Ferreira]

What I’m saying is that there are some ORM’s that provide numerous validation helpers. By performing validation on a previous stage, you are saying that you don’t want these validations.

Is this desirable? Maybe it is, in order to completely decouple yourself from the persistence layer.

You received this message because you are subscribed to a topic in the Google Groups "Clean Code Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clean-code-discussion/EyTIuGt_yTA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clean-code-discu...@googlegroups.com.

[James Green]

Convenience methods have been provided by a number of libraries/frameworks that do not meet with "clean architecture" definitions.

After all, move from JPA to MongoDB and be prepared to chuck your persistence tier away and with it your persistence-based validations.

[Luis Ferreira]

That was exactly my point.

So you’ll probably tend to having your own helper or convenience methods that are compliant with the clean architecture, right? Maybe even extract it into a library of sorts that is persistence . Does this make any sense?

[James Green]

The validation offered by persistence frameworks is perhaps only of use in very early prototyping. Here one might expect to expose an Entity to the public.

However an Entity's structure needs to be hidden from the public to allow for internal change. The need for validation therefore shifts to whatever accepts the data structures from the public.

Unfortunately the rise of persistence validation has led to a real surge in applications that do expose Entities beyond internal tiers. I'm as guilty as the rest, but I suspect the fact that I'm admitting to my mistakes means I'm learning.

The Java Validation API talks about applying rules to Business Objects (my capitalization) which in Java means Beans. In http://docs.oracle.com/javaee/6/tutorial/doc/gircz.html they talk about how this provides convenience between a web page input and those Business Objects. Those Business Object Beans will eventually lead to Entities in the persistence tier. That's my understanding.

[Sebastian Gozin]

I've also made the move to decouple input validation from persistence artifacts to well... input validation.
This mostly happened because I wanted to reuse the validation logic in my ATDD specs and if it was all tied to a specific persistence strategy that would've been impossible.

I'm use Grails GORM for a lot of persistence for example and I do like how they declare validation rules. I just don't like them on the actual GORM artifacts and instead want to apply them on arbitrary data structures. These could be regular object graphs or even a multidimensional map. What matters is the dynamic structure and their values not so much how they were created.

It was surprisingly easy to rebuild a GORM style validation DSL for use in my own code. It was basically a variant of Corey Haine's roman numeral kata with rules in place of numerals and a Groovy embedded DSL layer on top.

To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discussion+unsub...@googlegroups.com.

To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.

--
The only way to go fast is to go well.
---

You received this message because you are subscribed to a topic in the Google Groups "Clean Code Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clean-code-discussion/EyTIuGt_yTA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to clean-code-discussion+unsub...@googlegroups.com.

To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discussion+unsub...@googlegroups.com.

To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to a topic in the Google Groups "Clean Code Discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clean-code-discussion/EyTIuGt_yTA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to clean-code-discussion+unsub...@googlegroups.com.

To post to this group, send email to clean-code...@googlegroups.com.
Visit this group at http://groups.google.com/group/clean-code-discussion.

--
The only way to go fast is to go well.
---
You received this message because you are subscribed to the Google Groups "Clean Code Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clean-code-discussion+unsub...@googlegroups.com.