Re: ovdiscussion Re: Layering and packages

[David Allen]

Thanks for the thoughtful response, Mark.
Here are my reactions to your comments on the notion of having a single library for all shared code
First, we agree that such an arrangement is very much NOT an ideal design,
but only a transitional device to
allow me to refactor more readily while we consider a more ideal design.

You suggest that having less commonly used code in the main Library would make it larger and thus more difficult for newbies to learn. I agree with that general principle.
However, since a great deal of the code is commonly used by all the applications, I simply made the leap of reasoning that they would have to learn everything anyway.
But I will go back and closely examine the library and challenge my assumption to make sure that is true. In fact, I should include one of my colleagues in the process and see what their reaction is. Since I designed most of the classes in the shared code, I'm the least suitable person to judge how it looks to a newbie.

You've also described the risk in which libraries might use database tables would require every application to have tables that it doesn't use.
Can you explain this more fully?
It sounds like you're speaking from experience and I would like to understand the detailed scenario.
I was trying to imagine having some data access objects for example, which refer to several tables. But just because this library is referenced by an application, unless the application actually calls the data access objects which depend upon the tables, tables need not be created. I would expect my unit tests to uncover this immediately, during the earliest stages of the development process.
And if the application needs the data access object which needs the table, then I would simply create only those tables needed by the application. This seems appropriate.

But of course, I believe your deeper point was that we don't want to encumber applications which use our library with dependencies which are irrelevant to the application. And I admit that these external dependencies (like database tables) are some of the most annoying things to deal with. So as I consider this principle, I think about all of the third-party UI libraries and external API interfaces we depend on. I especially think about shared code which requires the special installation of software components. We have about five or six of these sorts of products. And while we expect our developers to have all of the products installed on their development machines so that they can support all of the applications and code, I can easily imagine scenarios in which we would like to test a small portion of the library on a machine without having to install a number of completely unrelated third party packages simply to satisfy the compile commands of a bloated library.
Anyway enough of this. The more I think about it, the more I realize the need to establish a rational package design scheme as soon as possible.
-----------
Regarding the book "Fundamentals of Object-Oriented Design in UML" by Meilior Page-Jones" :
He does have rules regarding how to use the layers. He agrees that packages in one layer should depend down. However I did not find any further rules on package design per-se. Most of his book is on class design.

Most of your criticisms of his model compared with the ARM are that the ARM is more specific, and thus is a stronger guide to better package factoring. In all fairness to Page-Jone's model, I just listed the top four layers -- but as you might imagine there is more to it than I can cover in a single email. Sometime over the next few weeks, when I have the time, I'll try to give you a more thoughtful reply to some of your specific concerns. But I will say off the top of my head that I like your separation of the user interface layer from the application layer.

Given your deep interest in package design and its relationship to software layers, I'm sure you would enjoy it tremendously. I got my copy on Amazon at a reduced price by buying a used copy. It must've been used in several courses as there are plenty of used copies available.
--------

So far, I have only found practical rules about software layers in your paper on The Architectural Reference Model (ARM) and Robert C. Martin's chapter on "Principles of Package Design" in his book "agile software development -- principles, patterns, and practices".
If you know of any other published writings on this topic, I would be grateful to have references.
Thanks again for your good ideas.
I may write again as I try to apply the ARM to my own project.
I have made a complete copy of one of our many applications along with all of the shared code and placed it in a sandbox where I can conduct all kinds of fun experiments with layering and package design.
I would love to attend your workshop, but I don't think my employer can afford to send me to Europe or afford to have you come here to the United States. However, if you ever do get an opportunity to present that or similar talks in the United States, please let me know and I will see whether I could make arrangements to attend.

Respectfully,
David Kreth Allen
Minneapolis, MN

[markcol...@gmail.com]

David,

 

Will give a more detailed response later, but RCM's CCP & CRP fit very nicely with the ARM. The ARM focuses on horizontal division of code into packages. Once that's been sorted, the CCP and the CRP can be used to subdivide each horizontal slice into packages.

 

Ciao,

Mark.
 

www.ratio.co.uk/markcollinscope.html
+44 (0)20 8579 7900 or :+44 (0)777 163 6882
Author:     "Agile Development with Iconix Process"
Editor:         www.ratio.co.uk/objectiveview.html
---------------------------------------------------------------------------
recursion is - of course - a form of magic

[markcol...@gmail.com]

David,

[more detailed reply as promised!]

Thanks for the thoughtful response, Mark.
Here are my reactions to your comments on the notion of having a single library for all shared code
First, we agree that such an arrangement is very much NOT an ideal design,
but only a transitional device to
allow me to refactor more readily while we consider a more ideal design.

You suggest that having less commonly used code in the main Library would make it larger and thus more difficult for newbies to learn. I agree with that general principle.
However, since a great deal of the code is commonly used by all the applications, I simply made the leap of reasoning that they would have to learn everything anyway.

It may be the case (I can't tell without some detailed information) that there really should be only one library. CRP states "package things together that are used together". If the same library API is necessary for 20 Apps, perhaps that's the right answer?

But I will go back and closely examine the library and challenge my assumption to make sure that is true. In fact, I should include one of my colleagues in the process and see what their reaction is. Since I designed most of the classes in the shared code, I'm the least suitable person to judge how it looks to a newbie.

You've also described the risk in which libraries might use database tables would require every application to have tables that it doesn't use.
Can you explain this more fully?
It sounds like you're speaking from experience and I would like to understand the detailed scenario.

It was purely and simply that because of a badly structured library, that had concrete DB related code (domain) and infrastructure code in it, we had to create endless unnecessary tables. This was a pain in the arse (ass) - excuse my French.

I was trying to imagine having some data access objects for example, which refer to several tables. But just because this library is referenced by an application, unless the application actually calls the data access objects which depend upon the tables, tables need not be created. I would expect my unit tests to uncover this immediately, during the earliest stages of the development process.

The library in question for my point required the tables to exist during application startup. So we had no choice but to create them. The other problem was that Domain packages were badly structured, so a "user login" library had links to a "members+" tables - which meant it couldn't be re-used without creating the members+ tables.
Live-able with, but certainly not ideal - "why do have to populate this table" was a common question. And with a bit of thought, this could have been structured correctly in the first place. We refactored it over time - which is the option you have. That works.

 

>And if the application needs the data access object which needs the table, then I would simply create only >those tables needed by the application. This seems appropriate.

In the short term, fine.

But of course, I believe your deeper point was that we don't want to encumber applications which use our library with dependencies which are irrelevant to the application. And I admit that these external dependencies (like database tables) are some of the most annoying things to deal with. So as I consider this principle, I think about all of the third-party UI libraries and external API interfaces we depend on. I especially think about shared code which requires the special installation of software components. We have about five or six of these sorts of products. And while we expect our developers to have all of the products installed on their development machines so that they can support all of the applications and code, I can easily imagine scenarios in which we would like to test a small portion of the library on a machine without having to install a number of completely unrelated third party packages simply to satisfy the compile commands of a bloated library.
Anyway enough of this. The more I think about it, the more I realize the need to establish a rational package design scheme as soon as possible.
-----------

Yeah. Refactor it. Make sure you have some automated tests to ensure you don't break something along the way. Do you have those?
 

Regarding the book "Fundamentals of Object-Oriented Design in UML" by Meilior Page-Jones" :
He does have rules regarding how to use the layers. He agrees that packages in one layer should depend down. However I did not find any further rules on package design per-se. Most of his book is on class design.

Remember I said I was second guessing. Could you send a PDF of the relevent pages - and I'll look at bit more at it. This shouldn't be a copyright issue as it will fall into "accepted usage" (in the UK at least).

Most of your criticisms of his model compared with the ARM are that the ARM is more specific, and thus is a stronger guide to better package factoring. In all fairness to Page-Jone's model, I just listed the top four layers -- but as you might imagine there is more to it than I can cover in a single email. Sometime over the next few weeks, when I have the time, I'll try to give you a more thoughtful reply to some of your specific concerns. But I will say off the top of my head that I like your separation of the user interface layer from the application layer.

Given your deep interest in package design and its relationship to software layers, I'm sure you would enjoy it tremendously. I got my copy on Amazon at a reduced price by buying a used copy. It must've been used in several courses as there are plenty of used copies available.

I'd need to see the relevent pages - don't want to say more when I haven't seen the base material.

>So far, I have only found practical rules about software layers in your paper on The Architectural Reference >Model (ARM) and Robert C. Martin's chapter on "Principles of Package Design" in his book "agile software >development -- principles, patterns, and practices".

This is a very good book taken as whole and I recommend it. Chapter 11 is a little confusing I think, and needs further explanation - it could be seen to contradict the ARM...  which of course can't be right... :-).

If you know of any other published writings on this topic, I would be grateful to have references.
Thanks again for your good ideas.
I may write again as I try to apply the ARM to my own project.
I have made a complete copy of one of our many applications along with all of the shared code and placed it in a sandbox where I can conduct all kinds of fun experiments with layering and package design.
I would love to attend your workshop, but I don't think my employer can afford to send me to Europe or afford to have you come here to the United States. However, if you ever do get an opportunity to present that or similar talks in the United States, please let me know and I will see whether I could make arrangements to attend.

Domain Driven Design - by Eric <forgot surname> shows a model very similar to the ARM. Well, almost identical actually. Craig Larman's book on UML and UP has a similar layering model.

Regards,
Mark.

[David Allen]

 

David Kreth AllenApplication DeveloperCarlson School of ManagementUniversity of MinnesotaMinneapolis, MN

> From: "m...@ratio.co.uk" markcol...@gmail.com

> Yeah. Refactor it. Make sure you have some automated tests to ensure you don't break something along the way. Do you have those?

 

Yes. There are automated tests for much of the code.

> Remember I said I was second guessing. Could you send a PDF of the relevent pages - and I'll look at bit more at it. This shouldn't be a copyright issue as it will fall into "accepted usage" (in the UK at least).

 

I'll see what I can do.

I can't find electronic copies anywhere so I must scan the pages first.

> >So far, I have only found ....Robert C. Martin's ..."agile software  development" ...

> This is a very good book taken as whole and I recommend it. Chapter 11 is a little confusing I think, and needs further explanation - it could be seen to contradict the ARM...  which of course can't be right... :-).

 

That may be one of my confusions.I notice his figure 11-2 shows lower layers depending on higher layers.

What is a poor little programmer like me to do when two "experts" have opposite opinions <smile>.

I guess I'll have to go back to the core principles (minimize duplication, minimize needless coupling, increase package cohesion) we all agree on and look at my unique situation.

> Domain Driven Design - by Eric <forgot surname> shows a model very similar to the ARM.

Eric Evans - and I've got it (shows you how well I read it <sheepish grin>).  I'll go back and take a look at it. I didn't understand it the first time, to tell you the truth.

 

> Well, almost identical actually. Craig Larman's book on UML and UP has a similar layering model.

I'll look at his book too. I've read some articles by him which I enjoyed.

thanks

[markcol...@gmail.com]

David,

> This is a very good book taken as whole and I recommend it. Chapter 11 is a little confusing I think, and needs further explanation - it could be seen to contradict the ARM...  which of course can't be right... :-).

 

That may be one of my confusions.I notice his figure 11-2 shows lower layers depending on higher layers.

What is a poor little programmer like me to do when two "experts" have opposite opinions <smile>.

I guess I'll have to go back to the core principles (minimize duplication, minimize needless coupling, increase package cohesion) we all agree on and look at my unique situation.

:-)!

I've had some discussions with Bob about this - though I don't think we reached any definate conclusions. One important point is that "Policy, Mechanism and Utility" are a little undefined - in that little explanation is given of what they actually mean (Martin Fowler uses these terms too). So it's unclear whether there is any mapping to the ARM layers.

However, assuming Utility is general purpose library stuff like Maths, then Mechanism cannot depend on it - not possible because the authors of Maths didn't know mechanism existed. Another problem is that Policy is most likely to change, so why is it at the bottom of the dependency chain. Breaks SDP imho.

What I do like about the packages he shows is that it is possible to make a particular package completely self contained, as in:

----------Policy Package--------------------------------
|PolicyCore|---------->|PolicyRequiredInterface|
----------Policy Package--------------------------------

In this case, PolicyCore calls PolicyRequiredInterface to access external services. This actually makes unit testing much easier, as the test code can provide a dummy TestPolicyRequiredImplementation which extends PolicyRequiredInterace. This has a nice feel about it to me.

Adopting this approach, I would then use Adapters/Proxys in a higher ARM layer to wire up PolicyRequiredInterface to an implemtation. Thus maintaing the rule that all dependencies point down:

E.g.                  ------------------------------------------------------|ApplicationAdapter|
                        |    (extends)                                                 |
                       V                                                                  | (calls)
|DomainPolicyRequiredInterface|                                            |
                                                                                          V
                                                                         |InfrastructureMechanism|

(ARM layer name precedes class name) Does that make sense?

Ciao,
Mark.
---------

[David Allen]

Yes, the diagram (Figure 11-2 in RCM's "Agile Software Development" is sparse on explanation for layering.

In fact, it's just plain vague. Well, the theory seems confident, but the class-like boxes are a bit ambiguous ("Mechanism Layer?", "Utility Layer")

Though most of the rest of the book is usefully precise.

So I am no longer concerned at the "apparent" contradiction between ARM and that particular example.

 

As a more practical test, I took a close look at one of my applications, and one

of my better uses of interfaces and the Dependency Inversion Principle.

 

And in my case, I tend to view the diagram with implementation at the same layer as the interface.

If I look at RCM's diagram, and move the implementers to the same level as the interface they implement,

then it is all "depends at the same layer or down".

 

Or the adapter mechanism you descrbied is more versatile when that is needed.

 

In either case, depending at the same layer or downward fits all the cases I've seen

of my own use of inheritance code so far.

 

Oh, and back to that comment of yours

"However, assuming Utility is general purpose library stuff like Maths, then Mechanism cannot depend on it"

 

Of course, you are right. But it clearly cannot be a "Maths" sort of thing. As that would make no sense at all, as you indicate. Instead, it implements the "Mechanism Service Layer" which to my mind means it does something on the order of complexity at least equal to the interface it implements, in which case it probably belongs in the layer above anyway.

 

So, yes, your example fits the ARM view, my experience also fits the ARM view, the diagram 11-2 does not ( but it is fine if you move the implementation classes to the same layer as the interface (or above it as you did in your example).

 

But enough on that.

I'm going to go test the ARM by moving my packages into those layers.

By the way, we are going to try a slightly different layering that looks like this

 

Interface

• see ARM

Application

• see ARM

Business

• see ARM

Platform

• what Meilir Page-Jones calls Architecture
• may include items from ARM Infrastructure and Platform

Foundation

• what Meilir Page-Jones calls Foundation
• may include items from ARM Infrastructure and Platform

 

As to "why" you probably already know why as you articulated it very nicely in

http://www.ratio.co.uk/W9.html

when you wrote

"We have chosen to separate the infrastructure and platform layers based on a buy versus build criteria. Architectural purists may object to this - why should the layer in which a particular component resides be dependent on whether you buy or build it? Our motivation is simple: we wanted to put the focus clearly on the aspects of the application being developed. For similar reasons we have been unconcerned with sub-layerings within the platform layer. "

 

Well, it's not the purchased classes we are interested in classifying,

but our own classes.

And just like you were unhappy with Meilir Page-Jone's "Application" layer because

the ARM split it into "Interface" and "Application" for better precision,

we like precision too. So we want to split what you call "Infrastructure" into "Platform and Foundation".

And see our definitions above as what we mean by platform is not exactly the same as what you call platform.

 

Anyway, we're just experimenting.

We'll let you know in a year how it went.

 

Ah the fun of being in a young industry.

That's why I love it so.

We get to invent new things.

 

Respectfully,

 

David Kreth Allen

Programmer

Carlson School of Management

University of Minnesota