Ultralight Components

[William la Forge]

When an object is built from a map, aggregating mixins is a trivial operation. But when a mixin needs to be closed, it is better to have a common mechanism to manage a stack of the close functions of all the aggregated mixins than to depend on application logic to do so. Case in point, I'd like to use an actor mixin or an alternative async channel mixin when composing a database. The async channel mixin will require a close but the actor mixin will not. So I developed the closer mixin to handle the calls to both the async channel and db file close functions. The database will only know the closer's do-close method. And it is the responsibility of the async channel and the db file mixins to register their close functions with the closer mixin.

My inspiration is Stuart Sierra's component library. But I am not handling dependencies per say, only managing a stack of close functions. And the "components" are only functions and atoms added to a common map structure, not records or deftypes. The result is a very lightweight pattern for components. The closer code itself is a lock-free mixin that gets added to the common map as needed when the on-close method is called.

For more information, see Closer.

(ns aatree.closer
  (:require [clojure.tools.logging :as log]))

(set! *warn-on-reflection* true)

(defn on-close [f opts]
  (let [fsa (:closer-fsa opts)]
    (if fsa
      (do
        (swap! fsa
               (fn [fs]
                 (if fs
                   (conj fs f)
                   (atom (list f)))))
        opts)
      (assoc opts :closer-fsa (atom (list f))))))

(defn- do-closer [fs opts]
  (when fs
    (try
      ((first fs) opts)
      (catch Exception e
        (log/warn e "exception on close")))
    (recur (next fs) opts)))

(defn do-close [opts]
  (let [fsa (:closer-fsa opts)]
    (if fsa
      (let [fs @fsa]
        (if fs
          (if (compare-and-set! fsa fs nil)
            (do-closer fs opts)
            (recur opts)))))))

[James Reeves]

Just to be clear, by "mixin" you're referring to merging two maps of functions?

Have you considered using composition instead of mixins? 

- James

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[William la Forge]

Sounds like you looked at wikipedia, which is the only place where I've seen mixin refer to the aggregate class. I've always seen mixin, or at least my understanding was that a mixin was the class that got added to the mixture. Could be just bad editing at the start of the wikipedia page, as they seem to reverse the sense of the term as you read further down the page. But as I look at other references to mixin, things become less and less clear.

Yeah, I am talking about creating compositions by aggregating the contents of other maps. It looks like a clearer term for the map being mixed into an aggregate would be trait. So I would have a db-file trait, the closer trait, the actor trait and the async-channel trait, all of which can be used in the composition of a database. Trait is clearer and nicely puts the emphasis on the functionality/capability being added.

It is hard coming up with good terms. I very much appreciate your help here James.

--b

[Raoul Duke]

re: mixins, traits, etc. those terms have all been used in both
research & shipped languages. Please see e.g. how Scala evolved with
those terms. :)

[James Reeves]

I didn't look at wikipedia, I was trying to guess what you meant from your post. I'm still unclear what you mean in the context of Clojure.

- James

[William la Forge]

James, when I used the term mixin I was referring to a map that acts like a trait that gets merged into a larger map. You would define several such smaller maps that can then be used in various combinations to compose "objects". The identity of the composite object (this) is the map which holds the merged contents of the smaller maps. I.E. The entries in the smaller maps get copied into the larger map. 

When executing functions held by a map, the last parameter is always the map itself, i.e. the "this". On the other hand, when placing closures into the map, the self reference is no longer needed as it is implicit in the closure. But this means that a closure can only reference the contents of the map when the closure was created, while a function can reference any of the contents of the map passed as its last argument.

Why did I make the map reference the last argument for functions held by the map? So that we can do type polymorphism on the first argument passed to the function. But we should make an exception to this. To facilitate threading, functions which return an updated map should take that map as the first argument. But that is an API change and needs to wait for release 0.6.0.

[Timothy Baldridge]

So I feel compelled at this point to ask..."why?". The whole point of functional programming in Clojure is to de-couple state from data. When you need polymorphic dispatch on the contents of a map, you have access to multi methods. Sure this is a fun thought experiment, but I don't understand the design goals. It's a fairly verbose way to write more complex code to accomplish what we already have good tools for (protocols/multimethods, etc).  Maybe I'm missing something. 

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Timothy Baldridge]

correction...."decouple state from behavior"

[William la Forge]

I think you have read too deeply into my thoughts on reserving the first argument of a function. I haven't actually written any polymorphic functions relating to this.

Really, the take off point for me is being able to operate on an object by implementing it as a map of functions and data. That is to say, making objects data. Implementing multiple inheritance becomes trivial and without having to define any classes or any interfaces. And with full support for circular references without needing to do declares.

What I like about it is that I get separation of concerns and maximal reuse without, I suspect, the usual usage coupling. The small maps which define traits can even participate in the lifecycle of the aggregate, so they start taking on the characteristics of components.

My biggest problem with writing code over the decades has been the constant desire to do rewrites--which are costly and devastating in their impact. That is why I am fascinated with this approach.

A second why is that when I have clear separation of concerns and the pieces of code can be easily tested independently, documentation becomes clearer and more fun to write. And keeping code fun is a critical driver for open source.

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/7Q7QvlSUGL4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[William la Forge]

Raoul,

The problem for me here is that all that terminology refers to classes. What I am dealing with here are objects implemented as persistent maps and operations on those maps. Which is why I'm grasping for terminology.

Perhaps I should just define a new term like UltraLight Component (ULC).

--b

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/7Q7QvlSUGL4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[James Reeves]

I feel you might be barking up the wrong tree with this approach, as it seems to complicate things without providing any compelling advantages.

That said, if it's fun then by all means continue to experiment. Maybe I'm wrong :)

- James

[William la Forge]

James,

Being fun is important but not a justification. We should strive to keep things fun, but there are plenty of thing worth doing and our resources are always limited. And if it is not fun, you are not going to do your best work.

Justifying things is generally impossible. If vanilla clojure meets your needs, then vanilla clojure it is. If macros solve the problems you have been dealt in the past, then three cheers for macros. If unifying client and server addresses your needs, then Om Next may well be a major blessing for you.

For me, the winner is avoiding static structures. I am tired of doing ongoing refactorings interrupted periodically by complete rewrites. Class hierarchies are the worst--being the largest, they are the least stable. I like small files that I can put to bed as finished. And self-defining aggregation. Because these meet my very real needs. I constantly reconceptualize what I am working on while bringing projects to completion. So having a programming style which keeps code relevant in the face of such an onslaught is very helpful and also a genuine delight.

--b

[James Reeves]

Have you watched Simple Made Easy?

I mention it because you remark about being tired of refactoring. It's my opinion that's a symptom of complexity, in the Hickey definition of the term.

- James

[Atamert Ölçgen]

Hi William,

How is this different than Yo-yo? (https://groups.google.com/forum/#!topic/clojure/PvCc5sSeSRY)

> Justifying things is generally impossible.

Actually, no. It is possible, at least when you are dealing with reasonable people. And when you make claims like "lightweight" or "ultralight", people will naturally ask "oh, interesting, how so?"

-----

Some functional programming folks are allergic to exceptions. They go out of their ways to prevent any and all exceptions and they end up unnecessarily complicating their types for little or no gain.

It seems to me, you are trying to avoid using protocols like it's a plague.

> I am tired of doing ongoing refactorings interrupted periodically by complete rewrites. Class hierarchies are the worst...

When your ultralight function based components get out of hand, you will have a worse time refactoring. With component you only have two functions, your lightweight design will end up having (* n 2) functions.

Also, looking at a component's code (it's defrecord form I mean) I can see what other components it depends on. Your design would bury them inside functions making it harder to read.

Kind Regards,
Atamert Ölçgen

◻◼◻

◻◻◼

◼◼◼

www.muhuk.com

[Colin Yates]

I think this discussion has run into the age-old problem of ‘subjective interpretation’, namely, multiple people all agreeing with a principle (i.e. complexity sucks) but then having _different_ interpretations of where that principle is challenged/upheld. Unfortunately in this industry there are hardly any unambiguous and objective statements because most of what we do is more artistic than engineering (zips up the flame suit now).

And, as ever, pragmatism throws it all out of the window :-)

One man’s complexity is another man’s simplicity and so on.

[Atamert Ölçgen]

Hi Colin,

> in this industry there are hardly any unambiguous and objective statements

Either you contradict yourself or the following is ambiguous and/or subjective:

> pragmatism throws it all out of the window

-----

> One man’s complexity is another man’s simplicity and so on.

Can you please give an example?

-----

Are you suggesting we don't discuss merits and shortcomings of different approaches to design software?

[Colin Yates]

Hi Atamert, good challenge.

TLDR: every choice is about weighing the pros and the cons and I am suggesting that the criteria we use is very very subjective and context sensitive

waffly comments inline.

> in this industry there are hardly any unambiguous and objective statements

Either you contradict yourself or the following is ambiguous and/or subjective:

Or both :-) I know! Isn’t this industry great :-)

I was too broad with my original statement - let me try again. I am suggesting that we are continually forced with making decisions/reasoning about things with very few objective and quantifiable metrics or tools to use to reason with. This leads to claims being made around something being ‘better’ or ‘worse’ without any agreed definition of the domain being discussed. Without any ubiquitous language if you will. Thus, many discussions lead to multiple inputs all claiming “this is right, that is wrong” but those choosing different things to be ‘right’ or ‘wrong’ about. And they are all right, and they are all wrong. They are all right for the person making the claim as these claims have to be context dependant because there are very few ‘global truths’ in this industry. They are (almost always) all wrong because they assert what is useful for them is useful for everybody.

Often what happens in these discussions is that somebody agrees with a principle (complecting is bad) and then proposes an approach which for them reduces it _for them_. Others, with different contexts and perspectives agree with the principle but see the proposed approach as either not reducing it or actually increasing it. At which point, until people are clear what their values are, the conversation ends at best with a "well chaps, let’s agree to disagree” or worse, people get offended. If people didn’t make such wide-ranging assertions then the discussion would be more fruitful.

> pragmatism throws it all out of the window

because "this is all well and good but I need to get on with stuff". I suggest that most of us continually live the dilemma of ‘do it now’/‘do it later’. A variation of which is ‘do it now with the tools I know about’/‘learn better tools and do it better, later’. 

Almost everything we (or at least I) do is a compromise of some sort because where the real world meets the computer is a mess of ambiguity, time pressure etc. So a programmer’s life is full of decisions which are trade-offs between choosing A and B. Static typing/dynamic typing, protocols/fns, which of the gazillion frameworks and libraries to use, kanban or sprint, unit testing? and so on and so on. 

-----

> One man’s complexity is another man’s simplicity and so on.

Can you please give an example?

Sure:

 - a big ball of mud is a single coherent domain model

 - micro services are nicely encapsulated and can introduce exponentially increasingly fragile deployment plans

 - design patterns are great and cause enormous bloat to a lot of code bases

 - dependency injection frameworks enable components to be ignorant and break encapsulation

 - static typing adds a whole level of safety and type signatures are bewildering

 - abstractions allow me to describe something I couldn’t before and add layer and layer of noise

 - lots of little namespaces keep things manageable and make navigation painful

 - lots of large namespaces keep things management and make navigation painful

 - DSLs bring capability to the user and add complexity to the code base

 - OO - what can I say here :-)

The point, the only point I am making is that all of the above is true depending on your view point because yes, complecting is bad, complexity is bad, and what makes something complex for me is different to what makes it complex to you. Where you see a number of things complected together I might see an example of encapsulation.

-----

Are you suggesting we don't discuss merits and shortcomings of different approaches to design software?

Not in any way, only that we explain the context in which we consider the approach ‘better’. I also wish we did discuss the merits and shortcomings, too frequently we don’t, we only discuss the merits.

The reason I posted initially is because I have wasted so many months arguing details with other developers and having a black and white view of things and it almost always turned out that we were agreeing with the general principle but the application of that principle meant different things in each of our different contexts. We were ‘violently agreeing’ :-).

TLDR(2): there is no ‘better’ only ‘better in this context'

I hope that clarified my view point.

[Herwig Hochleitner]

2015-11-24 4:31 GMT+01:00 William la Forge <lafo...@gmail.com>:

For me, the winner is avoiding static structures. I am tired of doing ongoing refactorings interrupted periodically by complete rewrites. Class hierarchies are the worst--being the largest, they are the least stable.

I don't think you'll gain much in that regard, by inventing a new way of pre-binding functions to data, a.k.a instantiating objects. The built-in mechanism for this is called clojure.core/partial. Much more can be gained by concentrating on the flow of data. Functions + immutable data are an excellent tool for that. In that image, an "object" would just be a (static) join point in the data-flow graph.

Nevertheless, there are legitimate use cases for composing behaviors mixin-style. Clojure directly supports this in the form clojure.core/extend, by which you can define a protocol implementation by a map of functions.

2015-11-24 9:11 GMT+01:00 Colin Yates <colin...@gmail.com>:

One man’s complexity is another man’s simplicity and so on.

More like: One man's state is another man's data, isn't it?

Seriously though, I'll have to insist that "simple" is an objective term, as opposed to the subjective "easy", hence assignments of simple vs complex should be agreeable. Thanks to Rich Hickey for that interpretation.

[Atamert Ölçgen]

> Not in any way, only that we explain the context in which we consider the approach ‘better’.

The context is the code William initially shared with the list.

I will not write a long response as I think this is getting off-topic for this list. You are very welcome ping me at my email if you would like to continue off-list.

[Colin Yates]

I’m not so sure I entirely agree :-) - complect/simple are unambiguous and can be measured, however there is still room for discussion as to _what_ is complected, but I think I am in danger of splitting hairs now.

[Colin Yates]

No, the content is all of William’s background, the lessons he has learned, the values he puts on various concepts, the needs he has etc.. For him, that code is ‘better’ because it addresses his needs. For others, with different needs, it might not be. And that is the point and why I injected my opinion, that talking in absolute terms isn’t entirely helpful.

William stated: "Justifying things is generally impossible.” which I thought was an interesting point given that is at the heart of this and many other discussions which take the form “I think this is better” and somebody else says “justify it” :-). I think it is possible if we share the rationale and context we are using to value these things.

I certainly didn’t mean to detract from the conversation, rather I wanted to keep it flowing as I think it is a good thread, but I could see it stalling, hence my injection.

[William la Forge]

James, thanks for recommending Simple Made Easy. As usual, I strongly agree with 95% or more of everything Rich says.

While listening to this talk, I kept thinking about why I am avoiding protocols. In general, I very much like having abstractions. But I find that even small abstractions tend to complect what with how.

Protocols are the perspective of the client code. It defines what is there to be used. But they should not be part of the code that provides a service. Rather, they should belong to the aggregate of service providers. Lets look at an example.

Suppose we have get and set! functions which an aggregate supports. Normally you would think of get and set! as being implemented by code that implements both. So you define a protocol that has both. It is a small protocol that has nothing else. But it is possible that these are implemented by very different pieces of code within an aggregate.

So it dawns on me. I need more code, not less! I should (a) define functions with the aggregate map as the first argument rather than the last and (b) use extend to add these functions to the Map interface. Now I have protocols. Only, there is no type name since all aggregates are maps. So I have not gained anything.

OK, so lets use deftype to define a type. It has a single field, which is the aggregate map. And we construct the type by creating that aggregate. AND THEN we extend that type with various protocols as convenient for how we use it. We need code to create specific aggregations anyway, so the only additional code here are the protocols and the extend. These provide the abstractions convenient to the code that wants to use the aggregate.

So wow, even more code. Have we accomplished anything??? YES! We have decoupled the abstractions from the implementations. Indeed, we can completely rework our abstractions (which I do a lot) without touching the code that implements them. For me, this is a major win. A substantial decomplection!

One question remains then. Is it worth the added code? This is simplification at the cost of easy. Loved that talk, James. :-)

[William la Forge]

So lets look at some artifacts then. I think we need an interface that all composites implement:

(definterface Composite

  (compositeMap []))

The function composite-map returns the map which holds the aggregated functions and data.

Now lets look at a sample function, blip:

(def blip [^Composite this x y z] ((:blip (.compositeMap this)) this x y z))

and a protocol that includes blip:

(defprotocol gran 

  (blip ^Composite this x y z))

Next we define a type which will hold an aggregate that includes an implementation of blip:

(deftype wackel [my-map]

  Composite

  (defn compositeMap [_] my-map))

  

and a function to create an instance of that type:

(defn new-wackel [this opts] (assoc opts :blip (fn [this x y z] (+ x y z)))

Finally, we can extend our type with the protocol:

(extend wackel 

  gran

  {:blip blip})

I find this uncomfortable. In Rich's terms this is simple but hard. To be expected, as it is a major decomplection. But it will take me a while to get comfortable with this. :-(

--b

  

  

[Timothy Baldridge]

So let's back up a bit and take a look at your assumptions. In your previous post you stated ", I kept thinking about why I am avoiding protocols. In general, I very much like having abstractions. But I find that even small abstractions tend to complect what with how." 

I would love to see an example of this in code. 

I tend to agree with some other people here, I think you're so far out in the weeds with this approach that you haven't realized that one of your starting assumptions could be incorrect. You state that protocols and multimethods complicate refactoring, can you show us an example?

Timothy

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[William la Forge]

It occurs to me that this can be simplified by using defrecord in place of deftype. We can eliminate the Composite interface, because compositeMap becomes the identity function. The code now becomes:

(def blip [this x y z] ((:blip this) this x y z))

(defprotocol gran 

  (blip this x y z))

(defrecord wackel)

(defn new-wackel [opts] (assoc opts :blip (fn [this x y z] (+ x y z)))

(extend wackel 

  gran

  {:blip blip})

This is better, but still a bit to swallow. Also, note I fixed a bug in new-wackel.

[William la Forge]

I better take this off line. I'm getting sloppy. Back to it later. 

[William la Forge]

Timothy, I'm definitely dancing on the leaves instead of the twigs. Or as you said, I'm far out in the weeds. As for the problems I've had, it is because I often change abstractions which have only minor impact on the code. Classically, I'll redefine the meaning of a value and need to make only one or two changes and get a major change in how a program works. Which is why I hate mixing comments in the code--the comments change a lot faster than the code for me.

The source of my divergence is my own mental processes. I do not visualize. Mostly I do not think using words. And I am a rabid first-principals thinker. So I am constantly re-conceptualizing.

I've gotten a lot of feedback. Very helpful. I really need to go quiet for a bit and digest all this.

Thanks!

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/7Q7QvlSUGL4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[James Reeves]

On 24 November 2015 at 13:48, William la Forge <lafo...@gmail.com> wrote:

James, thanks for recommending Simple Made Easy. As usual, I strongly agree with 95% or more of everything Rich says.

While listening to this talk, I kept thinking about why I am avoiding protocols. In general, I very much like having abstractions. But I find that even small abstractions tend to complect what with how.

One of the reasons Rich's definition of complexity appeals to me is because its a largely objective one. Complexity is a measurement of interconnectedness.

When we attach a protocol to a function, we're adding complexity, since there's now a connection where none existed before. We're essentially trading some additional complexity for a hopefully greater amount of functionality.

This implies that protocols should be used relatively rarely in Clojure, and my own experience seems to bear that out. In my projects directory, there are currently 2260 instances of defn, but only 35 instances of defprotocol.

 

Protocols are the perspective of the client code. It defines what is there to be used. But they should not be part of the code that provides a service. Rather, they should belong to the aggregate of service providers. Lets look at an example.

Suppose we have get and set! functions which an aggregate supports. Normally you would think of get and set! as being implemented by code that implements both. So you define a protocol that has both. It is a small protocol that has nothing else. But it is possible that these are implemented by very different pieces of code within an aggregate.

I don't understand why you'd want a protocol for that, nor what an "aggregate" is in this context. My suspicion is that there might be a simpler way of solving your problem without creating a new programming structure.

- James

[William la Forge]

James,

Much as I love stable code, I suspect that I am not there yet. Fortunately the code involved is still small. The feedback from you and others has been quite helpful. My preference of course is that we can stabilize it soon. :-)

--

[William la Forge]

James,

I've been thinking about protocols. Yes, I can now put them on an aggregate, but there may be a lot of places where it is better not to use a protocol and just use functions instead.

The code below does not use a protocol and I rather like it. In other cases we can use functions to achieve indirection by calling functions in an aggregate. So it remains to be seen if there is a good case for using a protocol with an aggregate. But at least I know how now--create an aggregate in a record.

Bill

(ns aatree.closer-trait

  (:require [clojure.tools.logging :as log]))

(set! *warn-on-reflection* true)

(defn on-close [this f]
  (let [fsa (:closer-fsa this)]

    (if fsa
      (do
        (swap! fsa
               (fn [fs]
                 (if fs
                   (conj fs f)
                   (atom (list f)))))

        this)
      (assoc this :closer-fsa (atom (list f))))))

(defn- do-closer [this fs]
  (when fs
    (try
      ((first fs) this)

      (catch Exception e
        (log/warn e "exception on close")))

    (recur this (next fs))))

(defn do-close [this]
  (let [fsa (:closer-fsa this)]

    (if fsa
      (let [fs @fsa]
        (if fs
          (if (compare-and-set! fsa fs nil)

            (do-closer this fs)
            (recur this)))))))

[Colin Yates]

<syntax-police>(let [x …] (if x true false)) is exactly what (if-let [x …] true false) is for (http://stackoverflow.com/questions/2010287/when-to-use-let-vs-if-let-in-clojure)</syntax-police>

:-)

[William la Forge]

Atamert,

I didn't get far on yo-yo--stopped reading when it started talking about monadic functions. :-)

But there is one main difference. I expect a component to "do its own thing", i.e. register its close function on opening. So there is nothing like defining a component. Well, except that when registering for a close, a component name is now being provided for use in the log. 

--b

[William la Forge]

+1