Composing Stuart Sierra's components

[Colin Yates]

I have a non-trivial component which requires a bunch of internal and external collaborators to work. This component is itself re-usable. 

What I really want to do is have ReusableComponent be a component in a system so it can pull its external collaborators. However, ReusableComponent constructs its own services etc. so it really want to be (or at least have access to) a system.

For example, let's say I have the following:

(defrecord InternalToReusableComponent [bus]

  (component/Lifecycle

    (start [this]...))

(defrecord ReusableComponent [bus logger]

  (component/Lifecycle

    (start [this]

      

      this)

    ....))

(defn reusable-component-system [external-collaborator]

  (component/system-map

    :bus (....)

    :logger (....)

    :reusable-component (component/using (map->ReusableComponent {}) [:bus :logger external-collaborator]))

Fine - I now have a system from which I can pull the reusable component. However, where does 'external-collaborator' come from? Obviously there is a larger system which I want this component to be part of so I can do:

(defn larger-system []

  (component/system-map

     :external-collaborator (...)

     :reusable-component (component/using (some-magic-glue) [:external-collaborator])))

I am struggling to see what (some-magic-glue) should be. I imagine it needs to be something like:

(defrecord SystemAdaptor [external-collaborator internal-system]

  component/Lifecycle

  (start [this]

    (let [internal-system (or internal-system (reusable-component-system external-collaborator))

           internal-system (component/start internal-system)]

     (assoc this :internal-system internal-system)))

  (stop [this]

    (let [internal-system (:internal-system this)

           internal-system (component/stop internal-system]

     (assoc this :internal-system internal-system)))

but it all feels a bit yuck.

I can't merge the two systems because the reusable component is chocka full of very fine grained command handlers and both the internal and external systems will have their own 'bus' for example. I could namespace the keys but that again feels painful...

Hope that is clear - and I look forward to your thoughts :).

[adrian...@mail.yu.edu]

You can specify component dependencies using the 'using' function as you know. As long as you know the key of the component in the system you can specify this dependency wherever you construct the component. If you want to parameterize dependencies, write a constructor function which takes the external dependency as a value. 

[Colin Yates]

Hi Adrian - I don't follow how that helps integrate two different
systems - I wonder if my question was unclear or I am missing
something in your answer. Would you mind posting some pseudo code to
clarify please?

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[adrian...@mail.yu.edu]

I believe I misunderstood your question; I didn't realize it was system (as opposed to any general component) specific. I think systems can be merged together (via 'merge'). Would that help? 

[Colin Yates]

merge won't help as there will be name space clashes.

I wonder if a more elegant approach would be to construct the 'inner'
system and then assoc onto it the external dependencies it needs
before calling start.

[Jonah Benton]

Hey Colin, it sounds like:

* if the 2 systems really can't function without each other, and their start/stop lifecycles are tightly bound, then somehow they have to be merged into a single system

or

* if the 2 systems can't be merged into a single system because of true functional or lifecycle independence and non-overlapping dependencies, then semantics for what it means for each system to not have the other available at lifecycle event times have to be established

It smells like the latter may be the way to go. So in pseudocode, the reusable-component-system that "owns" the reusable-component would not receive an instance of the external-collaborator at creation time. Instead, the reusable-component would have a somewhat weaker definition of its start semantics, and perhaps would have a way to refer to the external-collaborator only by name.

That is, perhaps the external-collaborator provided to the reusable-component is really an external-collaborator-client that is configured to know the name of the external-collaborator- which is owned by larger-system- and shields the reusable-component in cases where the larger system has not been started or the collaborator isn't available.

And perhaps similarly the larger-system has a reusable-component-client or reusable-client-proxy that is able to bridge appropriately from larger-system to the reusable-component.

And maybe both client components are configured with the same :bus for cross-system communication (and perhaps the cross-system bus is owned by a bridge system). Just riffing, as it's not precisely clear what the semantics of the systems are.

Stuart's comment that he hasn't run into a need for systems of systems is coming to mind. Perhaps it makes sense to break these apart more explicitly in accordance with guidelines around managing micro-services. Easy to say, of course. :)

Not sure if that's helpful....

Jonah

[Colin Yates]

Hi Jonah,

This is quite comparable to micro-services - each service is an abstraction or at least a facade and wants to play in a bigger system, but each micro-service may itself have its own stateful graph to maintain.

I think I will explore my original direction of having a AComponentWhichDrivesAnEntireSystem and see how far I get with that.

[Stuart Sierra]

On Wednesday, March 11, 2015, Colin Yates wrote:
> I can't merge the two systems because the reusable
> component is chocka full of very fine grained command
> handlers and both the internal and external systems will
> have their own 'bus' for example. I could namespace the
> keys but that again feels painful...

That's exactly how I would do it.

Nested systems don't really work.

But one system can contain implicit "groups" of components.

Create each group as a map, then `merge` them into the
system.

Use namespaced keys to prevent clashes.

The constructor functions for each component (or component
group) can take an argument which tells them the names of their
dependencies in the system.

Pseudo example:

    (defn reusable-component [collaborator-key]
      (component/using (map->ReusableComponent {})
        [:my/logger :my/bus collaborator-key]))

    (defn reusable-group [collaborator-key]
      {:my/logger ...
       :my/bus ...
       :my/reusable-component
         (reusable-component collaborator-key)})

    (defn new-system []
      (merge (system-map :main/logger ...
                         :main/bus ...
                         :main/collaborator ...)
             (reusable-group :main/collaborator)))


-S

[Colin Yates]

I like the idea of passing in the *key* of the external collaborator -
that's nice. Thanks Stuart.

I am surprised there isn't more call for nested systems - maybe there
is and this solution is sufficient-enough...

Again - thanks Stuart!

[James Gatannah]

On Thursday, March 12, 2015 at 10:16:10 AM UTC-5, Stuart Sierra wrote:

On Wednesday, March 11, 2015, Colin Yates wrote:

Nested systems don't really work.

FWIW, we've been using something that smells an awful lot like nested

systems for months now. I never realized we weren't supposed to.

Maybe we just got lucky?

- James

[Colin Yates]

Hi James,

Do you have a code fragment/gist for the glue?

[Stuart Sierra]

On Tue, Mar 17, 2015 at 5:47 PM, James Gatannah <james.g...@gmail.com> wrote:

FWIW, we've been using something that smells an awful lot like nested

systems for months now. I never realized we weren't supposed to.

It's not that nested systems never work, but from what I've seen they cause more complications than they're worth. The 'component' model doesn't forbid it, but it does not support dependencies between components in different "subsystems."

I've found it easier to keep system maps "flat" and use namespaced keywords to distinguish "subsystem" groups, even in large systems with 30+ components.

–S

[platon...@gmail.com]

I've also been investigating the nested system approach/problem.

The primary use case that I have is composing subsystems which are mostly independent modules using a higher order system to run in one process. The modules themselves can be easily extracted into separate applications thus becoming their own top level systems which makes it desirable to keep their system maps intact. 

Components inside modules might depend on the whole modules, not their constituent parts. This allows to have modules call each other through the API's in-process as well as being easily replaced by remote endpoints when separated into multiple processes.

This mostly works except for the components depending on other modules/systems, e.g.:

(require '[com.stuartsierra.component :as cmp])

(defrecord X [x started] 

   cmp/Lifecycle 

   (start [this] (if started (println "Already started " x) (println "Starting " x " " this)) (assoc this :started true)) 

   (stop [this] (println "Stopping " x " " this) this))

(def sys1 (cmp/system-map :x (cmp/using (X. "depends on dep" nil) [:dep])))

(def sys2 (cmp/system-map :y (cmp/using (X. "depends on sys1" nil) [:sys1])))

(def hsys (cmp/system-map :sys1 (cmp/using sys1 [:dep]), :sys2 (cmp/using sys2 [:sys1]) :dep (X. "dependency" nil)))

(cmp/start hsys)

Starting  dependency   #user.X{:x dependency, :started nil}

Already started  dependency

Starting  depends on dep   #user.X{:x depends on dep, :started nil, :dep #user.X{:x dependency, :started true}}

clojure.lang.ExceptionInfo: Error in component :sys2 in system com.stuartsierra.component.SystemMap calling #'com.stuartsierra.component/start

clojure.lang.ExceptionInfo: Missing dependency :dep of clojure.lang.Keyword expected in system at :dep

This happens because of the following:

1. Dependency :dep of sys1 is started in hsys

2. sys1 is started (:dep is started again, so the start/stop should be idempotent)

3. Dependency :sys1 of sys2 is started in hsys

4. :sys1 cannot be started as it depends on :dep which isn't present in sys2

This scenario could be supported by the Component library in several ways:

1. introduce an IdempotentLifecycle protocol which will be respected by the Component library. Implement the protocol for the SystemMap. IdempotentLifecycles will not be started or stopped for the second time, also their dependencies will not be updated if they are already started.

2. do not fail if a component already has a dependency under the specified key. This is a hack compared to the first solution, but I might go with it in the short term.

Stuart, what do you think about that? Would you consider a PR implementing the first proposal?

[platon...@gmail.com]

A possible implementation for the idea expressed in the previous post - https://github.com/stuartsierra/component/pull/25

[Dan Kee]

Sorry to resurrect an old thread with a somewhat tangential question but...

I'm seeing strange behavior in nesting systems that I am hoping someone can explain.  I have two independent systems as components of a super system, with an artificial dependency to attempt to enforce ordering of starting them.  When I start the super system, the first system starts, then the second, then first system is started again and I don't understand why.  I've since realized much simpler, more obvious ways to accomplish this, but I'd like to understand what I'm seeing.

Code (apologies for the macro, but it keeps things brief):

```

(ns component-debug                                           

  (:require [com.stuartsierra.component :as c]) )             

                                                              

(defn capitalize-symbol [s]                                   

  (symbol (clojure.string/capitalize (str s))) )              

                                                              

(defmacro make-example-component [name]                       

  (let [capitalized-name (capitalize-symbol name)]            

    `(do                                                      

       (defrecord ~capitalized-name []                        

         c/Lifecycle                                          

         (start [~name]                                       

           (println (str "Starting " '~name))                 

           (assoc ~name :started true) )                      

                                                              

         (stop [~name]                                        

           (println (str "Stopping " '~name))                 

           (assoc ~name :started false) ) )                   

                                                              

       (defn ~(symbol (str "new-" name)) []                   

         (~(symbol (str "map->" capitalized-name)) {}) ) ) ) )

                                                              

(make-example-component foo)                                  

(make-example-component bar)                                  

(make-example-component baz)                                  

(make-example-component qux)                                  

                                                              

(defn new-system []                                           

  (c/system-map                                               

   :foo (new-foo)                                             

   :bar (c/using                                              

         (new-bar)                                            

         [:foo] ) ) )                                         

                                                              

(defn new-system2 []                                          

  (c/system-map                                               

   :baz (new-baz)                                             

   :qux (c/using                                              

         (new-qux)                                            

         [:baz] ) ) )                                         

                                                              

(defn new-super-system []                                     

  (c/system-map                                               

   :system (new-system)                                       

   :system2 (c/using                                          

             (new-system2)                                    

             [:system] ) ) )                                  

                                                              

(defn -main [& args]                                          

  (c/start (new-super-system)) )                              

```

Output:

```

Starting foo

Starting bar

Starting baz

Starting qux

Starting foo

Starting bar

```

Thank you!

--Dan

[Stuart Sierra]

Hi Dan,

The key to understanding what's happening here is to
remember that `component/start` combines both dependency
ordering *and* dependency injection.

Your "super system" looks like this just after it is
constructed:

    {:system {:foo {}, :bar {}},
     :system2 {:baz {}, :qux {}}}

When you call `component/start` on this system, it
dispatches to `start-system`, which will build the
dependency graph to see that :system must be started before
:system2.

Next, start-system will call `component/start` on the
component at :system. That component is itself a SystemMap,
so it dispatches to `start-system`, which starts :foo and
:bar.

Now it's time to start :system2. start-system sees that
:system2 has a dependency (`component/using`) on :system. So
it will `assoc` :system into the component at :system2. Now
the system looks like this:

    {:system {:foo {:started true},
              :bar {:started true,
                    :foo {:started true}}},
     :system2 {:baz {},
               :qux {},
               :system {:foo {:started true},
                        :bar {:started true,
                              :foo {:started true}}}}}

Notice that :system2 now contains a complete copy of
:system.

The rest should be obvious. Starting :system2 will start
:baz, :qux, *and* the nested copy of :system. So :system
gets started again. :foo and :bar were already started, but
`component/start` doesn't know that, so it starts them
again.

This is another reason I don't recommend nesting systems:
the behavior is not obvious unless you deeply understand the
model.

There are 2 ways to prevent the repeated starting of :foo
and :bar in this example.

1. Define a custom record for your "super system"
   implementing component/Lifecycle in a way that knows how
   to start the subsystems in the correct order without
   `assoc`ing their dependencies.

2. Define the `start` and `stop` methods of :foo and :bar to
   check if they have already been started before starting
   them again. (i.e. make them idempotent)

-S

[Dan Kee]

Ah, that makes sense.  Thanks for the thorough response!