Decomplecting "component"

[Anatoly]

I looked around for good patterns, and found Malcolm's code examples from the real project (I believe it was "Open Sensors") really useful. I still have an uneasy feeling about a "component" framework, since it:

• steals the fun from Clojure
  
• too verbose
  
• repetitive
  
• not really beautiful Clojure code
  
• destroys the Clojure approach of "libraries over frameworks"
  
• leaves more questions than problems it solves
  

Having said that, I do understand that state needs to be managed somewhere, and component seems to do it well. Hence I'd like to ask a few questions to start a discussion (and to have a better understanding in the process):

1. What would be your preferred grouping of state and behavior? i.e. say I have 3 sources: DB, Amazon SQS, some external REST thingy. I would create 3 components to hide and manage their lifecycle (i.e. state). But what about functions that deal with this state? For example would you have a (receive-messages-from-sqs [...]) loop in the same namespace as the Amazon SQS component?

2. Would you group components into a single namespace (i.e. system "A"), or would you rather have a single namespace for each?

3. Let’s say there are two components A and B. B depends on A, and has a function that needs to communicate something to A. With all the examples it looks like B would pass A to that (B) function. In my mind it couples A and B quite tightly. Would it not make more sense to only pass core.async channels to all the components, and have communication between ALL the components be via these channels? (e.g. a component C would have a channel "ch-a" as its state, B and A would be injected with this "ch-a", and when a function in B wanted to pass something to A, it would just put that something onto "ch-a")

4. Following up from the question #3: Can a component then just be three things: an atom, an input channel, an output channel? Just thinking out loud, since I have not used "component" yet.

5. I would also be very interested to learn what did not work for you with using "component", and what you would recommend to do right from the start that will help later on.

Thank you,
/Anatoly

[Juan @tangrammer]

Hi Anatoly! 

As I (fortunately) worked  with juxt and specifically in opensensors.io project during a few months, I think I can share here my  component and juxt/modular experience/ideas and questions

On one side, the development of new features or extending functionalities using modular/components (or maybe ... could we call them "modules"?) and component/reloaded workflow was really enjoyable (clear final design included). 

When I met stuartsierra/component, and having a bit of experience using java dependency injection frameworks, I also mixed this component perspective with "SpringFramework way of thinking" and deriving that... it "steals (a bit) the fun from Clojure and not really beautiful (and creative) code" or in other words it seemed to me like a Java (Bored) Pattern

But this disappointment and the idea about the java relation were disappearing as far as I worked in solving the co-dependency (bidirectional dependency) or customising 1,2,3 stuartsierra/component (BTW: great and beautiful stuart code!), and at the same time getting familiar with juxt/modular ideas (thanks to Malcolm patient) about generating component systems with data/configuration using tested modules, lein templates and edn data (great clojure code here too!). 

And in relation to the title "Decomplecting component" and/or

5. I would also be very interested to learn what did not work for you with using "component", and what you would recommend to do right from the start that will help later on.

For me, the problem is that: it is very easy to mix the idea of component as "Managed lifecycle of stateful objects" with "Encapsulate/Hide state into objects(defrecords)". Or maybe this problem is derived that I cant' answer the question: "why do we need to hide the system state to the components of this system?"

If we follow same pattern as Om, so there is a global app/system state and the components use it (cursors), we can use stuartsierra/component only for start/stop the global app/system state and passing the full system to normal clojure functions (no protocols). My "secret" ideal would be passing this full system to prismatic/graphs but still no real code around it :) 

I hope my comments were useful and luckily this discussion keeps growing

Cheers

Juan

[Anatoly]

Hi Juan,

Thanks for sharing! I did not reply right away since I wanted to soak in component for a bit to get a feel for it before forming an opinion.

Using component for almost two weeks :) I still feel it sucks you in to the framework rather than give you functional freedom where everything is based on small composable functions. I do see the benefit of using component with Stuart's tools.namespace, but having done projects without both, I am still skeptical about a full "buy in".

The current project, I use component for, does have many external sources, hence lots of connection / cache state that I need to keep and manage somewhere, and for now I am going to keep leaning on component to do that, but it still feels like early Spring days where the inversion of control was "the thing". Back in 2005+ it made a lot of sense, since there was no good way to manage stateless, stand alone, components that interact with state (in JVM world), and Spring solved it really well.

Ten years later (now), I feel like component drags me back to "trust me to control it for you" which splits the beautiful universe of just functions to 2: just functions and (now) classes. They are classes with fields (keys) and behavior (lifecycle + functions under the keys). This creates a mental gap and makes me reason about different pieces of code differently.

I do understand that Juxt is based on component, and there is no going back, since you already invest so much into "modular". And I believe it is a good way, I just question whether it is the way for all Clojure projects that deal with state (talking about projects for clients: i.e. not generic libraries).

For example, take Scala. I am not a big fan: http://www.dotkam.com/2013/10/29/scala-where-ingenuity-lies/ but I did some projects where I dealt with state without frameworks. It varied, I still had lifecycle, but most of the state was transformational (functions passing state to each other, actors, etc..).

I am still looking for a good way to manage "component state" in Clojure, and I believe it should be done without buying into the framework (which component is). Actually that is what prompt me to do more Reagent and less Om, since Reagent is not a framework, but Om kind of is.

Your links are really helpful, I'll definitely try to visualize as much as I can. I am thinking of https://github.com/MichaelDrogalis/dire for AOP since it is just function based.

Thank you,

/Anatoly

[Martin Trojer]

Hi Anatoly,

thanks for sharing your view, and I do agree with a lot of your points. I'd like to explain the non-lispy 'component situation' that Clojure has ended up in with a short history lesson.

Clojure is hosted language, and JVM is the most popular host (at the moment). This gives Clojure a lot a libraries / tooling for free, but also some drawbacks. One of the biggest one for Clojure/JVM is the (lets be frank) abysmal REPL launch time, especially when you have more than a handful dependencies. Since almost all Clojure jars contains source, we basically have to compile the project and all its dependencies every time we launch the REPL.

Compared to other lisps this is a pretty significant difference (REPL launch times of several minutes vs micro seconds). Clojure developers answer is to never 'bounce the REPL', and in order to make this plausible we need a way of resetting our environment (including reading new namespaces etc etc). This is what clojure.tools.namespace solves and why it was adopted so quickly.

Using c.t.n. leads to other problems, namely process-level resources like ports, db connection pools etc. If we don't lifecycle those, reset will fail! A couple of solutions came about to solve this, Stuart Sierra made the 'reloaded pattern' which eventually turned into component. 

Some time have now passed, and new Clojure developers who doesn't have the historical context are looking at Component and seeing Spring. They are also seeing classes/objects/members/methods in the lifecycle records. They get excited about it, and this is why 'stuart sierra component' is mentioned in every other Clojure conference talk.

My point is that we have been driven to c.t.n and component, not just because its merits but to overcome a pretty serious limitation in the combination of the Clojure compiler and the JVM.

[Malcolm Sparks]

Everything Martin says is true, but there is also another (critical) area that component addresses, which is to provide an out-of-band lexical scope for passing state to implementation code.

See Herwig's useful explanation of this here: https://groups.google.com/d/msg/clojure/xxEK9rmKeSk/2iN8pvTfQNUJ

I have spoken about this too, but only have video format: https://skillsmatter.com/skillscasts/5820-components-with-clojure

One of the reasons I was attracted to component (not just clojure.tools.namespace) was this feature, because I've seen what happens to large(ish) clojure code-bases without it:-

1. Function arities drift upwards, causing breakages and brittleness

2. Ad-hoc architecture degrades, losing consistency

3. Multiple competing approaches to this problem (of passing state) get blended together

An example of 3 is where people have this habit of hijacking the Ring request (which of course only works for the web portion of application), and 'injecting' things like database connections into Ring requests, which is a bit like having to hitch-hike your way into work every day - it 'works' but it just feels wrong. Since I've started using component consistently, I've never had to resort to this kind of thing.

I have to dash so will stop here, but there are plenty of good points in Anatoly's post which are worthy of further discussion.

--
You received this message because you are subscribed to the Google Groups "modularity" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modularity+...@googlegroups.com.
To post to this group, send email to modul...@googlegroups.com.
Visit this group at http://groups.google.com/group/modularity.

To view this discussion on the web, visit https://groups.google.com/d/msgid/modularity/cc73ceaa-e5fc-4794-97ce-ddbdb1d43a03%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Martin Trojer]

I agree with you Malcolm, I am weary about people taking this Spring-thing way-way too far, see these 2 particularly horrifying examples.

• https://tbaldridge.pivotshare.com/media/testing-episode-1/22222/feature
• https://www.youtube.com/watch?v=av9Xi6CNqq4&spfreload=10
  

There is hardly any 'Clojure left' when the component framework has totally devoured your codebase.

[Anatoly]

  The history makes sense, and I watched it happened, but never liked the solution :)

  I remember Stuart's early component presentation at NYC Clojure meetup, where I had this mixed feeling of "yes, it solves it" and "no, I don't like this solution, it steals from me".

  Again, while I do understand where you guys are coming from, and given the fact that for one of my current projects I decided to stick with component (mostly because it has about 20 external sources to deal with), I am not arguing that this is a wrong solution. I just want to use as least as possible of it, while focusing the majority of my codebase on "functions we love".

  It is interesting that people embrace component to the point where there is no Clojure left, it is probably an easy trap to get in to, especially when you start testing: "oh.. look, if it was also a component, I could have just merge a stub instead.."

  I like the idea of bound lexical scope to decrease function arities and to (visually) group related things together. So maybe I can meet this solution in the middle, where I define my own protocols that deal with state and have lifecycle (which is just another protocol). No libraries, no frameworks, just protocols to scope the state and functions for everything else. The remaining problem is dependency graph.. that would potentially be defined using a map: {:a [:b :c] :b [:d :e] :c [:e] ...} (nothing else: just a map of protocol keys). Just thinking out loud. c.t.n would still work, since everything that needs to be cleaned up would by lifecycable.

  Overall it seems that as long as I can program (reset) to reset the external state, using c.t.n, I may completely avoid component ('like) rigidity. As to function arities and lexical scoping, again thinking outloud.. why can't functions that take lots of state be refactored to several functions that encapsulate that state:

(not real/working code, just making it up as I type)

(defn f [work-with state]

  (fn [& data]

    (work-with data state))) 

(def find-user (f user-by-id datomic))

(def add-order (f store-order datomic))

(def send-email (f email mailer))

(def create-receipt (f post-receipt http-client))

(def find-receipt (f get-receipt http-client))

;; and now..

(defn create-order [user-id item-id]

  (-> (find-user user-id)

      (add-order item-id)

      (create-receipt user-id)))

;; or..

(defn send-confirmation [user-id order-id]

  (let [user (find-user user-id)]

    (->> (find-receipt order-id)

         (send-email user))))

/Anatoly