separation of concerns w/o encapsulation

[Brian Craft]

Talk on the list about encapsulation usually comes back to some variation of "you don't need it when you have immutable data structures". But in the long term I'm finding the problem of working w/o encapsulation is not the danger of data being mutated under you. Rather, it's the danger of all the module boundaries blurring over time, leading to the big ball of mud: a very fragile code base where everything depends on everything else.

E.g. if you model your application with a plain data structure which you pass around to different modules, each concerned with a small part of that data structure, the tendency over time is for every module to become concerned with every part of that data structure.  Then you have no separation, nothing is reusable, and the code is very fragile.

How do you avoid this, practically? In OO it would be enforced with encapsulation, so the next developer (which might be me, six months later) trying to fix a bug, or add a feature, knows "Oh, I shouldn't peek in here: this module isn't supposed to depend on that piece of data".

[James Reeves]

Avoiding encapsulation doesn't mean that every function needs to act on a huge global data structure. On the contrary, one of the core ideas of Clojure is that we should try to reduce incidental interconnections. What you describe in your example is essentially complexity creep.

Ideally, functions should access only the data they need. For instance, you might have a function:

    (defn available [{:keys [stock reserved]}]

      (- stock reserved))

In the function arguments we're deliberately restricting the data down to two keys. We could also use core.typed to enforce this for us:

    (ann available ['{:stock Int, :reserved Int} -> Int])

- James

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Fingerhut]

James, are you saying that with such a core.typed annotation for function available in that example, that it would give a warning if available or any of the functions it calls accesses any keys other than :stock and :reserved ?

If so, that is quite impressive.

Andy

[James Reeves]

On 8 May 2015 at 18:40, Andy Fingerhut <andy.fi...@gmail.com> wrote:

James, are you saying that with such a core.typed annotation for function available in that example, that it would give a warning if available or any of the functions it calls accesses any keys other than :stock and :reserved ?

Yes and no. The function can access other keys, but their type will be "Any", limiting their use.

For instance, if I changed the function to include an additional key, without changing the type annotation:

    (defn available [{:keys [stock reserved pending-delivery]}]

      (- stock reserved pending-delivery))

Then core.typed would raise a type error, since [Int Int Any] is not a valid type signature for clojure.core/-.

However, it might be worth raising the suggestion to Ambrose to add a way of failing immediately if an unknown key is used, rather than just assigning the "Any" type.

- James

[Stuart Sierra]

I find it's really the same as in any other language. Certainly if you don't have any clearly-defined boundaries at all, you'll get a big ball of mud.

Encapsulation is about code organization and self-discipline. Define module responsibilities and boundaries in your developer documentation. Make it clear that X is not supposed to depend on Y. Enforce those boundaries through code review and refactoring. Regularly review module definitions to make sure they match the real requirements of the system.

I developed Component[1] to help with one aspect of this problem. One shared map defines the structure of the entire system, but each “module” is only exposed to the subset of the system it needs.

Other approaches: With a shared map, namespaced keywords can be a hint that something is “private” to a particular module. Alternately, you could establish the convention that elements of a shared data structure should *only* be accessed via helper functions, and use public/private Vars to enforce which aspects of a data structure are meant to be “public” to other modules.

–S

[1]: https://github.com/stuartsierra/component

[Frank Castellucci]

Brian

One is better off thinking that avoidance, as per your note, is a discipline of better practices. As the architectural concepts of Separation of Concerns and Minimized Surface Areas were intended.

Many languages attempt to enforce this notion in something called encapsulation. However; Encapsulation is buzz as it is subject to the pressures of the moment, including but not limited to:

1. Market pressure for timely delivery - or "Ef it, I'm just going to expose this one little thing for now..."
2. Introspection - If I can find it, I'll figure out how to get/set it. Especially easy through such things as:
3. Byte Code Injection - In the case of Java, and,
4. Easy work arounds - In the case of Clojure's  (def my-local #'ns/their-privates)

How to avoid it? Discipline and code reviews.

Frank Castellucci

[Gary Verhaegen]

If your modules are becoming big and unwieldy, you can also take a step toward (conceptual) OO and have your modules be separate entities that communicate through message passing (i.e. objects, although probably closer to Erlang's processes than Java objects). Messages are the key missing piece of many OO languages; Clojure maps are really nice for that. You can use core.async to establish communications between your modules, and defining the interfaces becomes akin to defining a network protocol (though with richly structured messages).

This way, each module is only exposed to what other modules send it, your code is well decoupled and isolated, and when it keeps growing and it makes sense to separate the code base into different processes, the refactoring is, if not trivial, manageable.

Disclaimer: this is just an idea, I have not had the opportunity to actually try it out on a real, reasonably complex codebase.

--

[Brian Craft]

Thanks, everyone, this has been very helpful.

[Georgi Danov]

No tool or technology beats the combination of:
a) component/responsibility blueprint
b) discipline in communicating, following and adapting it

encapsulation can be a nice safety net once you have the things above, but it would never be a solution to the problem. just a convenience.

coding and designing defensively because you are concerned about your teammates is big waste of time. if this is the reality (in the enterprise it's the norm) guess clojure is not the best tech for that organization.

[Brian Craft]

On Tuesday, May 12, 2015 at 4:29:22 AM UTC-7, Georgi Danov wrote:

coding and designing defensively because you are concerned about your teammates is big waste of time. if this is the reality (in the enterprise it's the norm) 

Yes, it is the norm in the enterprise. In a decade of enterprise coding, encapsulation became a conditioned response. Anything exposed would be abused, guaranteed. It was impossible to enforce through documentation, due to the size of the organization. Requiring another team to read the docs, and respect API boundaries when committing, involved trying to persuade three layers of management that it was important enough to institute such a policy. From the VP perspective, shipping next week was always more important. Encapsulation was the only tool that worked.

To some degree, it's the norm in other contexts as well. We see in popular languages w/o encapsulation that users invariably code to the internals, making it difficult or impossible to refactor libraries.

http://ziade.org/2010/03/03/the-fate-of-distutils-pycon-summit-packaging-sprint-detailed-report/

When you have enough users, the cost of such breakage is quite high. Then "they should have known better" doesn't fly as an argument.

[Juan A. Ruz @tangrammer]

Hi guys, 

when you talk about encapsulation do you mean using defrecords + protocols ?

In this case, we are talking of choosing defrecords instead of plain functions. Maybe it would be better if we talk too about the drawbacks of this choice. For example, can we compose or extend protocol functions (I mean to extend existent component protocol functions). 

In some occasion, I wanted to use juxt/bolt [(formerly juxt/cylon) security component based system] in GoogleAppEngine but juxt/bolt uses http-kit and GAE dont' let you use this lib dependency. So at last I had to fork juxt/bolt to switch the http client library for plain java.net classes. So, I wonder myself which are the limits of reusability in component pattern. 

[Brian Craft]

http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_programming)#As_information_hiding_mechanism

[Sean Corfield]

On May 13, 2015, at 11:07 PM, Brian Craft <craft...@gmail.com> wrote:

http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_programming)#As_information_hiding_mechanism

In an environment where you can’t trust your co-workers(!), you certainly want to hide *mutable* data so they can’t modify your objects’ state. When you remove mutability, there’s a lot less damage they can do and they it’s really more about which pieces of the system do you not want them to "use" (in terms of functional APIs).

The "Clojure Way" seems to be "just make all functions public — they might be useful to someone" although that then restricts what refactoring you can do in future compared to exposing a smaller surface area. Since you can always call private functions anyway (and access private data directly), you really need conventions — and code review — to prevent abuse, and at that point having foo.api and foo.impl namespaces — and a convention to not use foo.impl outside of foo.api — is very reasonable and even automatically enforceable through tooling.

When I first started doing Clojure (five years ago — yikes! feels like yesterday!) I still had an OOP mindset around data hiding and I used defn- and ^:private a lot (well, ^{:private true} back then I guess). These days I tend to make all my functions public and use namespaces if necessary to clearly delineate APIs vs implementations. A def of immutable data tends to be public too. It’s rare that I feel the need to make things private since immutability means no one can "abuse" my data or my functions.

Sean Corfield -- (904) 302-SEAN

An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Colin Yates]

I guess a related concern is abstraction. I notice I often have functions which work at different levels of abstraction in the same ns which makes me uncomfortable. In OO land they would be package level or even instance classes. I haven't yet found a way to solve this in clojure land.

To be explicit, I have a defn which calls a defn which calls a defn which calls ...., having all of those defns public doesn't capture the hierarchy of abstraction.

[Sean Corfield]

On May 14, 2015, at 10:28 AM, Colin Yates <colin...@gmail.com> wrote:
> I guess a related concern is abstraction. I notice I often have functions which work at different levels of abstraction in the same ns which makes me uncomfortable. In OO land they would be package level or even instance classes. I haven't yet found a way to solve this in clojure land.
> To be explicit, I have a defn which calls a defn which calls a defn which calls ...., having all of those defns public doesn't capture the hierarchy of abstraction.

…although it would allow other users of your code to build their own abstraction hierarchies. They can’t do that if you make things private "arbitrarily".

If you really want to separate them and provide strong guidance that there is an intended hierarchy of abstractions, put them in different namespaces — named to indicate the layers in your hierarchy — and leave them all public.

That creates a much more reusable, extensible code base. IMO (now — I didn’t think that way five years ago).

[Colin Yates]

That assumes the intermediate functions are reusable. I guess with all these things asthetics come into play, and there is of course the option of letfn as well.

[Erik Price]

On Thu, May 14, 2015 at 12:14 PM, Sean Corfield <se...@corfield.org> wrote:

In an environment where you can’t trust your co-workers(!), you certainly want to hide *mutable* data so they can’t modify your objects’ state. When you remove mutability, there’s a lot less damage they can do and they it’s really more about which pieces of the system do you not want them to "use" (in terms of functional APIs).

The code itself is mutable. The nice thing about a private function is that I have reserved the right to change how it works someday, without fear that others have become dependent on the way it currently works.

e

[Sean Corfield]

On May 14, 2015, at 12:37 PM, Erik Price <er...@zensight.co> wrote:
> The code itself is mutable. The nice thing about a private function is that I have reserved the right to change how it works someday, without fear that others have become dependent on the way it currently works.

That’s why I suggested foo.api and foo.impl namespaces and requiring (convention, peer review) that nothing outside of foo.api uses the foo.impl namespace — any exceptions would need to be specifically discussed and sanctioned.

And remember there’s a difference between building a library for "the public" to use and building software for internal use by your own team or other peer teams in your division.

[JUAN ANTONIO RUZ]

pasting here what I think could be a separation of concerns code pattern proposal from "Graph: Abstractions for Structured Computation"

... Object-oriented programming encourages the decomposition of large systems into objects that encapsulate their state, making it easier to build, test, and reason about these systems. OOP was a massive step forward in software engineering, and today nearly all large-scale software endeavors (video games, search engines, etc.) are built within this paradigm.

The reason for this shift is that the older generation of procedural code with no state boundaries came with a complexity overhead that made it harder to understand the code than the actual concepts being represented. No software abstraction can reduce the inherent complexity of a problem, but it can reduce the complexity overhead, making it possible for engineers to build complex systems faster and with fewer bugs.

...

Once an engineer comes to grok FP, they tend to organize code around how data 'flows' between these pure functions to produce output data. The structure of how functions connect to form the structure of a functional computation has typically been informal. Until now....

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/mJqQJn_oW4A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Raoul Duke]

> Once an engineer comes to grok FP, they tend to organize code around how
> data 'flows' between these pure functions to produce output data. The
> structure of how functions connect to form the structure of a functional
> computation has typically been informal. Until now....

see Flow Based Programming, Dataflow, LabView, etc.