[ANN] Introducing Yo-yo, a protocol-less, function composition-based alternative to Component

[James Henderson]

Hi all,

I've just released an early version of 'Yo-yo', a protocol-less, function composition-based alternative to Component. It's still in its early stages, so feedback would be very much appreciated!

https://github.com/james-henderson/yoyo

Yo-yo was also an experiment to see what could be de-coupled from the concept of 'reloadable systems', so you won't find any configuration, dependency injection, etc - just a way to write a system that can be easily started, stopped, and reloaded.

Fundamentally, we start by assuming there's a function available that only returns 'when the system stops' - a 'latch', say. If we had such a function, we could start our system, call that function, then stop the system (closing any necessary resources). A database pool, for example, might look like this:

(defn with-db-pool [db-config f]
  (let [db-pool (start-pool! db-config)]
    (try
      (f db-pool)

      (finally
        (stop-pool! db-pool)))))

Here, we're assuming that we'll be passed 'f', the 'latch' function. A web server would be similar, and, because they're both functions, they're very simple to compose:

(with-db-pool {...}
  (fn [db-pool]
    (with-web-server {:handler (make-handler {:db-pool db-pool})
                      :port ...}
      (fn [web-server]
        ;; TODO: Ah. We've run out of turtles. :(
        ))))

This is where Yo-yo comes in - there’s a function called run-system!, which takes a function that accepts a latch:

(:require [yoyo])

(yoyo/run-system!
  (fn [latch]
    (with-db-pool {...}
      (fn [db-pool]
        (with-web-server {:handler (make-handler {:db-pool db-pool}) ; n.b. we have access to the db-pool here - no need for global state!
                          :port ...}
          (fn [web-server]
            (latch))))))) ; Aha!

run-system! then returns a promise - deliver any value to it, and it'll stop the system.

And that's pretty much it! There are a few more functions - mostly to do with easily starting/stopping/reloading a system through the REPL, and a macro to simplify the 'function staircase' - these are covered in more detail in the README. There are some also common components - a database pool, a web server, and a simple integration for existing Component systems.

It'd be great to hear your thoughts/ideas, whatever they may be - either through here, e-mail, Github, or Twitter - thanks!

James

[Atamert Ölçgen]

Hi James,

Interesting idea. Thanks for sharing.

I think you can simplify this:

(yoyo/run-system!

 (fn [latch]

   (ylet [db-pool (with-db-pool {...})

          :let [server-opts {:handler (make-handler {:db-pool db-pool})

                             :port 3000}]

          web-server (with-web-server server-opts)]

     (do-this web-server)

     (do-that db-pool web-server)

     (latch))))

to:

(yoyo/foo! [db-pool (with-db-pool {...})

            :let [server-opts {:handler (make-handler {:db-pool db-pool})

                               :port 3000}]

            web-server (with-web-server server-opts)]

  (do-this web-server)

  (do-that db-pool web-server))

I believe with-* function can also be simplified further.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Kind Regards,
Atamert Ölçgen

◻◼◻

◻◻◼

◼◼◼

www.muhuk.com

[James Henderson]

Hi Atamert - thanks :)

I thought it might be preferable to keep the call to (latch)explicit - it means that ylet can be used in nested calls, too - for example, to set up and compose groups of components/sub-systems: (contrived example, though!)

;; (docs for ylet at https://github.com/james-henderson/yoyo#introducing-ylet )

(require '[yoyo :refer [ylet]])
 
(defn with-connections [config f]
  (ylet [db-pool (with-db-pool (:db config))
         es-conn (with-es-connection (:elasticsearch config))]
 
    (f {:db-pool db-pool
        :es-conn es-conn})))
 
(defn make-system [latch]
  (let [config ...]
    (ylet [connections (with-connections system)
           _ (with-webserver {:handler (make-handler (merge connections
                                                            {:config config}))
                              :port 3000})]
      (latch))))

How would you see the with-* functions working, btw?

Cheers,

James

[Atamert Ölçgen]

On Tue, Jun 23, 2015 at 11:47 PM, James Henderson <ja...@jarohen.me.uk> wrote:

Hi Atamert - thanks :)

I thought it might be preferable to keep the call to (latch)explicit - it means that ylet can be used in nested calls, too - for example, to set up and compose groups of components/sub-systems: (contrived example, though!)

;; (docs for ylet at https://github.com/james-henderson/yoyo#introducing-ylet )

(require '[yoyo :refer [ylet]])
 
(defn with-connections [config f]
  (ylet [db-pool (with-db-pool (:db config))
         es-conn (with-es-connection (:elasticsearch config))]
 
    (f {:db-pool db-pool
        :es-conn es-conn})))
 
(defn make-system [latch]
  (let [config ...]
    (ylet [connections (with-connections system)
           _ (with-webserver {:handler (make-handler (merge connections
                                                            {:config config}))
                              :port 3000})]
      (latch))))

How would you see the with-* functions working, btw?

I think the general idea should be to provide a clean API to the consumer (of your lib). Perhaps something that accepts a start function, a stop function and some sort of main loop (f in your example).

[James Henderson]

On Wednesday, 24 June 2015 11:17:41 UTC+1, Atamert Ölçgen wrote:

On Tue, Jun 23, 2015 at 11:47 PM, James Henderson <ja...@jarohen.me.uk> wrote:

Hi Atamert - thanks :)

I thought it might be preferable to keep the call to (latch)explicit - it means that ylet can be used in nested calls, too - for example, to set up and compose groups of components/sub-systems: (contrived example, though!)

;; (docs for ylet at https://github.com/james-henderson/yoyo#introducing-ylet )

(require '[yoyo :refer [ylet]])
 
(defn with-connections [config f]
  (ylet [db-pool (with-db-pool (:db config))
         es-conn (with-es-connection (:elasticsearch config))]
 
    (f {:db-pool db-pool
        :es-conn es-conn})))
 
(defn make-system [latch]
  (let [config ...]
    (ylet [connections (with-connections system)
           _ (with-webserver {:handler (make-handler (merge connections
                                                            {:config config}))
                              :port 3000})]
      (latch))))

How would you see the with-* functions working, btw?

I think the general idea should be to provide a clean API to the consumer (of your lib). Perhaps something that accepts a start function, a stop function and some sort of main loop (f in your example).

Not sure I understand what you mean here? Tbh, I was trying to get away from the idea of separate start & stop functions - it seems 'cleaner' to me without them! (although of course that's subjective). 

Also, the 'with-*' functions here are consumer code - the only Yo-yo functions/macros in this example are 'run-system!' and 'ylet'. Yo-yo itself is tiny (<100 LoC) - my aim was for a library that solely dealt with starting/stopping a provided system, and no more :)

Maybe it'd be worth fleshing out an example of what you were looking for?

Cheers,

James

[James Henderson]

Seems like the next step for this would be for me to put together a blog with an example Component system, and its equivalent Yoyo system?! :) Should have time for that over the weekend.

James

[James Henderson]

As promised, have blogged: 'Yo-yo & Component - Side by Side'

Contents:

• Making components
• Using a component as a dependency
• Serving a REST API
• Wiring it all up
• Yo-yo / Component Interoperability
• Mockable Services
• ‘Mocking out’ dependencies

Let me know what you think!

Cheers,

James

[Timothy Baldridge]

A few bits of feedback after seeing these examples. 

Firstly, I'd like to see a more fleshed-out rationale. Currently it sounds a bit like cargo-culting, e.g. "Spring is bad, Component reminds people of Spring, therefore Component is bad". I'd challenge several parts of that rationale, but the first is that "spring is bad". Why is that so? Is it the XML configuration? Because Component doesn't have that. Is it the mutability? Because Component doesn't have that either. Sadly I'm afraid some people seem to see polymorphic functions (protocols) and think "AAAHHH OOP! OOP IS BAD!!". When that's not really the case. The worst parts of OOP are mutability and encapsulation of state inside opaque objects. Component (and clojure for that matter) discourages both of these, so I'm not sure I see the problem. 

Secondly, I'm a bit leery of using anonymous functions instead of records. I once worked on a system with a co-worker, and I asked him "why are you using a record here...why not a closure?". He replied: "Because I can't see what's inside a closure...it's opaque". That bit of advice has stuck with me for some time. With something like component, if I have a function that takes a component, I can look at that argument and see something like this:

=> #db.DBClient {:server "127.0.0.1" :port 4242 :schema "my-schema"} 

With Closures I get something like this:

=> <fn foo.bar.baz >

That doesn't help much with debugging. 

With Records I get immutability plus a type I can extend to any protocols. Also, since my protocol is clearly defined, it's simple to extend. I don't have to worry about hidden functions some inner function may call on my client. Protocols provide abstraction. 

So I guess that's my critique of Yo-Yo. I'd love to see a more in-depth rationale, and I get nervous when people replace protocols with plain functions, because normally I loose some expressiveness in the process. 

Timothy 

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Thomas Heller]

Hey,

interesting approach but I don't like the nesting and "manual" wiring of dependencies. I don't quite like that every with-* function remains on the stack as well, but it shouldn't hurt that much. An uncaught exception will also take down your entire system, but I guess you'd have a try/catch in your "latch" anyways.

But what I miss the most is an instance of your "app" (ie. all components together). You create it yourself in the example but I really want that always. Sometimes you just want to access your system from the outside just to see whats up (eg. REPL into a live system). I also consider the webserver to be a "client" of my "app" and not part of it (or another layer of it if you will), but that is a topic for another day.

Way way back in the day I used to work with (and on) PicoContainer which was/is a dependency injection and lifecycle management container. I tried writing a DSL for it (in Groovy, this was 2003 or so) but concluded that Java already was good enough to set everything up, a DSL (or XML) is overkill. All you need to describe a "Component" is:

a) what are its dependencies

b) how do I start it

c) how do I stop it

In that light I wrote my own "dependency injection" helper functions since nothing like Stuart's Component existed at the time I got into Clojure. I don't like Component due to its invasive protocol but in essence I do the same.

In my system I just set up a map of components and use that as a descriptor for wiring:

{:a {:depends-on []

     :start my.components.a/start

     :stop my.components.a/stop}

 :b {:depends-on [:a]

     :start my.components.b/start

     :stop my.components.b/stop}}

     

The key in the outer map becomes whatever the :start function returns and is refered to it by its name :a (the key of the map). The :start function of :b is called as (my.components.b/start instance-of-a). An instance of a component is treated as an opaque value and other components interact with it only via its "public" interface (ie. my.components.a). Whether this is done via a protocol or not doesn't matter. When a shutdown is requested the :stop function is called with the instance of the component as the argument.

That is about it. Mocking is just assoc over default descriptor map and I have helper functions to only do partial start/stop calls if only a specific component is needed (eg. I only need :a).

Like I said it basically does the same stuff as Component, just a little less invasive since I think a component should not know about the container it runs in.

  

Hope that was somewhat useful as feedback to Yo-Yo.

Cheers,

/thomas

[James Henderson]

Hi Timothy - firstly, thank you for taking the time to read through and reply. I understand it would have been very easy to read them and move on thinking 'just a nutter on the mailing list who doesn't really understand what he's talking about', but I do really appreciate your feedback - thanks :)

You're right - openly, I think it took not only hammocking about Yo-yo, but also sitting down over the weekend writing up the side-by-side to really realise what Component and Yo-yo both provide. (I don't regret doing either, of course - without doing so I wouldn't have got close to thinking about it in this way!) I do agree that not everything about Spring/OOP is bad, and certainly that Clojure/Component has taken the good parts. What I've seen so far, though, is some people (me included, probably) seeing Component, thinking 'that's nice, a pattern I'm familiar with', and reverting to writing Clojure apps like they used to write Spring apps - with components for *everything*, components for wiring up other components, etc, giving them all sorts of names (a la 'the Kingdom of Nouns'). It's also Phoenix, in particular, that reminded people of Spring - I've tried to ensure I always make that distinction, apologies if that didn't come across.

Incidentally, I wrote a first iteration of the side-by-side blog with the Component side written in this style, and then realised there was nothing in Component that mandated writing in that way; just that I and others had interpreted it that way - it was quite an 'aha' moment! I'd also not heard that advice about anonymous functions either - thanks! Thinking about it, I've come up with all sorts of workarounds to get closed-over variables out of closures.

Having said that, I think there's still some legs in Yo-yo - given that it doesn't mandate writing apps in any style (only that the one top level function accepts a system latch) it's possible to write Yo-yo apps as you would a Component system, or using functional composition, or anywhere in between. At the very least, Yo-yo would help with managing Component's system at the REPL - the top-level function becomes:

(defn make-system [f]
  (let [started-system (-> (c/system-map ...)
                           c/start-system)]
    (try
      (f started-system)
     
      (finally
        (c/stop-system started-system)))))

(defn -main [& args]
  (yoyo/set-system-fn! 'myapp.main/make-system)
  (yoyo/start!))

with the ability to then run (yoyo/reload!) etc from the REPL (without everyone having to write their own user namespace start/stop functions)

I do also like the way that there aren't separate start/stop functions - maybe there's scope for a Lifecycle protocol that only involves one function instead? Maybe that's not even a good idea either, I don't know. I'll keep on experimenting, that's for sure :)

Thanks again,

James

[James Henderson]

Hey Thomas, thanks for your e-mail :)

On Monday, 29 June 2015 11:25:44 UTC+1, Thomas Heller wrote:

Hey,

interesting approach but I don't like the nesting and "manual" wiring of dependencies.

I've found people at both ends of that particular spectrum - some that won't live with DI, some that won't live without it :) I guess a library like Yo-yo has two options - either be opinionated about it, or let people choose one or the other. In this case, I've chosen to let people choose - there's nothing about Yo-yo that mandates the nesting (except the top-level function) - what you do within that is up to you.

 

I don't quite like that every with-* function remains on the stack as well, but it shouldn't hurt that much.

Hmm - I was wondering about that too. Maybe an approach similar to trampoline would help here?

 

An uncaught exception will also take down your entire system, but I guess you'd have a try/catch in your "latch" anyways.

I'm not sure it will? If there's an exception thrown during system startup, the components will then have an opportunity to stop themselves (in the reverse order) because of their try/finally's - I'd say this is the behaviour we'd want, in order to avoid half-started systems. Once the system's started, and (latch) called, an uncaught exception in the components won't stop the system - because it'll be thrown on a different thread, if I understand correctly? Certainly need to write some test cases around it!

 

But what I miss the most is an instance of your "app" (ie. all components together). You create it yourself in the example but I really want that always. Sometimes you just want to access your system from the outside just to see whats up (eg. REPL into a live system). I also consider the webserver to be a "client" of my "app" and not part of it (or another layer of it if you will), but that is a topic for another day.

Yep, I agree with this - I've been using some workarounds to get values out of the system, none of them particularly pretty. Interesting idea about the webserver being a client of the app - would be good to see where you take that?

Way way back in the day I used to work with (and on) PicoContainer which was/is a dependency injection and lifecycle management container. I tried writing a DSL for it (in Groovy, this was 2003 or so) but concluded that Java already was good enough to set everything up, a DSL (or XML) is overkill. All you need to describe a "Component" is:

a) what are its dependencies

b) how do I start it

c) how do I stop it

In that light I wrote my own "dependency injection" helper functions since nothing like Stuart's Component existed at the time I got into Clojure. I don't like Component due to its invasive protocol but in essence I do the same.

In my system I just set up a map of components and use that as a descriptor for wiring:

{:a {:depends-on []

     :start my.components.a/start

     :stop my.components.a/stop}

 :b {:depends-on [:a]

     :start my.components.b/start

     :stop my.components.b/stop}}

     

The key in the outer map becomes whatever the :start function returns and is refered to it by its name :a (the key of the map). The :start function of :b is called as (my.components.b/start instance-of-a). An instance of a component is treated as an opaque value and other components interact with it only via its "public" interface (ie. my.components.a). Whether this is done via a protocol or not doesn't matter. When a shutdown is requested the :stop function is called with the instance of the component as the argument.

That is about it. Mocking is just assoc over default descriptor map and I have helper functions to only do partial start/stop calls if only a specific component is needed (eg. I only need :a).

Like I said it basically does the same stuff as Component, just a little less invasive since I think a component should not know about the container it runs in.

Looks another interesting approach :) I'm currently hacking on some similar ideas myself - think there's plenty of room for iteration in this area at the moment!

 

Hope that was somewhat useful as feedback to Yo-Yo.

Certainly was! Thanks! :)

[Thomas Heller]

Hey James,

"the webserver being a client" is really very simple. Basically instead of starting one "app" you start two. Your actual "app" and the "web-app" that depends on "app". One contains your business logic and the other everything related to translating HTTP to app API calls. "app" doesn't know about the web part.

The component stuff also assumes that the web server itself is not a component. A typical Servlet Container is built this way, it assumes that it will host your app and not the other way around. I use http-kit but I still want that hierarchy. Trying to turn the web server itself into component just produces nightmares of cyclic dependencies and such.

Can't really explain this very well, it is way too hot to think straight.

I wanted to create a proper library out of my component stuff for a while but never get around to it. I might do that some day to create an actual example I can refer to.

Cheers,

/thomas

[Atamert Ölçgen]

On Fri, Jul 3, 2015 at 2:19 PM, Thomas Heller <th.h...@gmail.com> wrote:

Hey James,

"the webserver being a client" is really very simple. Basically instead of starting one "app" you start two. Your actual "app" and the "web-app" that depends on "app". One contains your business logic and the other everything related to translating HTTP to app API calls. "app" doesn't know about the web part.

Seems like you are describing hexagonal architecture. (https://blog.8thlight.com/uncle-bob/2012/08/13/the-clean-architecture.html)

[Thomas Heller]

That article makes it sound like an OOP beast, it is really much simpler than that.