[ANN] components.md

[Renzo Borgatti]

Hello clj,

kinda late into that discussion of a while ago about “how I use components”, I finally wrote-up about it:

https://github.com/reborg/scccw/blob/master/COMPONENTS.md

The result is a markdown document that is meant to be used by copy-pasting. So no libraries, no frameworks. The reason is that I’m mainly introducing an architectural convention to handle stateful parts of a Clojure application and I believe your app should bootstrap from these general principles and extend on them, evolutionary. It’s not different from saying that you should isolate impure functions in your code: you wouldn’t create a library out of it.

Let me know what you think. Between this organization for components and Fluorine for config (https://github.com/reborg/fluorine/), I’m a happier Clojure dev these days :)

Renzo
@reborg

[James Reeves]

Why implement your own Lifecycle protocol, rather than using the one from the Component library?

- James

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[rebo...@gmail.com]

Not sure it's worth to depend on a library for a defprotocol with two functions. Also, still need to double check, but I think I can get rid of that defprotocol.

Renzo

[James Reeves]

On 5 April 2016 at 17:23, <rebo...@gmail.com> wrote:

Not sure it's worth to depend on a library for a defprotocol with two functions. Also, still need to double check, but I think I can get rid of that defprotocol.

Well, the advantage with using the Lifecycle protocol from Component is that you can make use of existing libraries.

- James

[Timothy Baldridge]

The thing that this doc also advocates (and I am against) is manual dependency management, your app start function must call start on its dependencies. This not only hardcodes the dependency, but also requires the user to think about the startup order of components on every usage. 

In addition this approach doesn't support substituting alternate implementations. During testing I often want to use a in-memory queue vs RabbitMQ. Or perhaps I want to wire an entire system up with in-memory queues to perform limited integration testing. Hardcoding dependencies limits the flexibility of the app. 


And this part of the doc:

"A: You see, this all idea of reusable libraries of components never worked and it gets complicated pretty fast. See the past 20 years of OO frameworks. Do you really want a Spring/J2EE in your beautiful Clojure app?"

Is just wrong. Perhaps across multiple projects and multiple companies component re-use isn't as common as some would think. but within the same company/project components are re-used a lot. Not only does this quote it present a false dichotomy (either you have a nice Clojure app or you have reusable components). But it is also little more than an ad-homiem attack: "This looks like OOP, OOP is bad, therefore doing this is bad!"

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Renzo Borgatti]

Thanks Tim for reading down till the last line!

> On 5 Apr 2016, at 19:09, Timothy Baldridge <tbald...@gmail.com> wrote:
>
> The thing that this doc also advocates (and I am against) is manual dependency management, your app start function must call start on its dependencies. This not only hardcodes the dependency, but also requires the user to think about the startup order of components on every usage.

This “every usage” is roughly every time a new component is added (in my experience). Of the total number of components, around 80% is added at the very beginning and a few more during the lifetime of the app. Last time I had the ordering problem with one component I fixed it in the app start function and never touched it again. So I’m not sure if the user is required any special effort after the very first setup.

> In addition this approach doesn't support substituting alternate implementations. During testing I often want to use a in-memory queue vs RabbitMQ. Or perhaps I want to wire an entire system up with in-memory queues to perform limited integration testing. Hardcoding dependencies limits the flexibility of the app.

Good point about stubbing out sometimes. But shouldn’t something like (with-redefs [bootstrap/system {:rabbit “my stub”}] <test-here>) solve the problem easily? As of running the app with a different configuration, you could get away by “TESTING=true java -jar target/uberapp.jar” and put a conditional in system.clj to wire-up the app with in-memory queues.

> And this part of the doc:
>
> "A: You see, this all idea of reusable libraries of components never worked and it gets complicated pretty fast. See the past 20 years of OO frameworks. Do you really want a Spring/J2EE in your beautiful Clojure app?"
>

> Is just wrong. Perhaps across multiple projects and multiple companies component re-use isn't as common as some would think. but within the same company/project components are re-used a lot. Not only does this quote it present a false dichotomy (either you have a nice Clojure app or you have reusable components). But it is also little more than an ad-homiem attack: "This looks like OOP, OOP is bad, therefore doing this is bad!”

You are right, I’ll re-phrase that to be more precise: "This looks like Java, Java is bad, therefore doing this is bad!”

What I’m trying to say is that I’m not big fan of libraries of components for clojure: https://github.com/danielsz/system or https://modularity.org/ comes to mind. Those are meant to be cross-organization. Even internally tho, I don’t like to build and grow our own library of components. At the beginning it sounds like a good idea, in the long term it doesn’t scale. There are several reasons that doesn’t work well:

1. one project need component #1 of X in the library and suddenly you find yourself with ActiveMQ client (and many others) in your classpath when you don’t use them. You can deal with it at the cost of an increase of the complexity of the library of components.
2. if internal library, it requires additional release cycle effort: an internal repo to host the lib, a snapshot/concrete builds management, in general doing changes in two projects at once.
3. if a project requires the component to act differently (say I need elasticsearch native client but all the rest can stay on the REST client) I bump the component library version +1 and add some config. Now on, I need to remember why there is a version difference and to pay attention when I update other projects. Still something you can handle with some additional effort.

So I noticed that if I just copy paste what I need in each project, it works as intended and scales just fine.

Now more on the Java bashing. My thinking is the following: if a language ecosystem constantly pushes you to depend on frameworks to maintain sanity developing an app there is something wrong. I would have some hard time convincing a Java team to just use JDBC and a db driver to connect to a DB. So we’ll use Spring and whatnot. In Clojure we have the luxury that the language doesn’t need framework patching. So I think I shouldn’t have said OOP, but more precisely Java.

Thanks
Renzo

[Timothy Baldridge]

Thanks for replying. 

>> (with-redefs [bootstrap/system {:rabbit “my stub”}] <test-here>) 

That would possibly do the trick, but I really don't like using redefs, it assumes a mutable environment, and then also locks me into a single-threaded testing system, one where all uses of a var must refer to the same thing. In short, I don't want to configure my system using code, I want my system to be configured with data. Onyx is a good example of this, define components and name them, then wire them up together with data. Not only does this allow for programatic configuration of a system, but also programatic introspection (https://github.com/walmartlabs/system-viz). 

>> What I’m trying to say is that I’m not big fan of libraries of components for clojure

I completely agree there. Such libraries have very limited use.

>> I would have some hard time convincing a Java team to just use JDBC and a db driver to connect to a DB. So we’ll use Spring and whatnot. In Clojure we have the luxury that the language doesn’t need framework patching. So I think I shouldn’t have said OOP, but more precisely Java

I like the way a co-worker put it once: "Ask yourself, will I ever need more than one of these?". Will I ever need more than one database connection? Will I ever need more than one queue? Will I ever need more than one image server, etc. I'd say that 50% of the time the answer is "yes" and another 49% of the time you are in denial and will say "yes" sometime in the future :). In that case I'd like my system to talk via a common interface that is fairly easy to swap out, perhaps even on a sub-project basis.

Now these components are often very application specific, so I don't advocate some central library of components, but with a single company there could be dozens of components that are all wired together to compose the systems that create the application as a whole. 

So that's what I have worked with for years now at some fairly large companies, and I have found to give me the biggest "bang for the buck": Non-singleton components that express themselves via a well defined abstract protocol, wired together via data. 

[Jeroen van Dijk]

> The thing that this doc also advocates (and I am against) is manual dependency management, your app start function must call start on its dependencies. This not only hardcodes the dependency, but also requires the user to think about the startup order of components on every usage.

This “every usage” is roughly every time a new component is added (in my experience). Of the total number of components, around 80% is added at the very beginning and a few more during the lifetime of the app. Last time I had the ordering problem with one component I fixed it in the app start function and never touched it again. So I’m not sure if the user is required any special effort after the very first setup.

At my company we *do* have complicated dependency trees that you really don't want to manage manually or even think about. We've build a layer on top of Component actually. We use edn config files to declare what components should be in the system. The dependencies are calculated accordingly (I hope we can open source it some time)

 

> In addition this approach doesn't support substituting alternate implementations. During testing I often want to use a in-memory queue vs RabbitMQ. Or perhaps I want to wire an entire system up with in-memory queues to perform limited integration testing. Hardcoding dependencies limits the flexibility of the app.

Good point about stubbing out sometimes. But shouldn’t something like (with-redefs [bootstrap/system {:rabbit “my stub”}] <test-here>) solve the problem easily? As of running the app with a different configuration, you could get away by “TESTING=true java -jar target/uberapp.jar” and put a conditional in system.clj to wire-up the app with in-memory queues.

With the configuration files for instance, we substitute a Kafka component with an in-memory logger component during testing and a file-based logger in development. I have also been able to generate a full cluster of 10 nodes of the same server component in the same JVM running on 10 different ports. This was possible because the configuration is just edn data and we don't have any global state in our components [1]. This is also something that wouldn't be possible with Mount as this library seems to promote global state.

[1] except for the system var of course

[Renzo Borgatti]

> On 7 Apr 2016, at 13:42, Jeroen van Dijk <jeroentj...@gmail.com> wrote:
>
>
> > The thing that this doc also advocates (and I am against) is manual dependency management, your app start function must call start on its dependencies. This not only hardcodes the dependency, but also requires the user to think about the startup order of components on every usage.
>
> This “every usage” is roughly every time a new component is added (in my experience). Of the total number of components, around 80% is added at the very beginning and a few more during the lifetime of the app. Last time I had the ordering problem with one component I fixed it in the app start function and never touched it again. So I’m not sure if the user is required any special effort after the very first setup.
>
> At my company we *do* have complicated dependency trees that you really don't want to manage manually or even think about. We've build a layer on top of Component actually. We use edn config files to declare what components should be in the system. The dependencies are calculated accordingly (I hope we can open source it some time)

Could you give a couple of examples? I just can’t imagine how bad it can get. In my case, apart from a bus initialization requiring a couple of components to be initialized first, everything else is very linear. But of course, I might just be lucky.

Also, are we using the same definition of component? I tend to identify a namespace “n" as a component if reloading n or its dependents has the potential of putting the application in an incongruent state (that is, something that you can’t possibly reach while the application is running normally). def or defonce with IO or atoms are usually a good indicator.

> > In addition this approach doesn't support substituting alternate implementations. During testing I often want to use a in-memory queue vs RabbitMQ. Or perhaps I want to wire an entire system up with in-memory queues to perform limited integration testing. Hardcoding dependencies limits the flexibility of the app.
>
> Good point about stubbing out sometimes. But shouldn’t something like (with-redefs [bootstrap/system {:rabbit “my stub”}] <test-here>) solve the problem easily? As of running the app with a different configuration, you could get away by “TESTING=true java -jar target/uberapp.jar” and put a conditional in system.clj to wire-up the app with in-memory queues.
>
> With the configuration files for instance, we substitute a Kafka component with an in-memory logger component during testing and a file-based logger in development. I have also been able to generate a full cluster of 10 nodes of the same server component in the same JVM running on 10 different ports. This was possible because the configuration is just edn data and we don't have any global state in our components [1]. This is also something that wouldn't be possible with Mount as this library seems to promote global state.

I don’t know Mount sufficiently to judge, but I’m not promoting global state either. But there instances where a namespace needs a (def cache (atom [])) for example. The important thing is that I can control the reload of that “def” from a centralized place (a very handy one being user/reset).

Renzo

>
> [1] except for the system var of course
>

[William la Forge]

Ah, discussions which do not define terms are subject to endless round and rounds.

I see several characteristics of a component. The first is that there is a lifecycle. A component may need to be opened and closed.

The second is that components are not singletons. So you can have multiple instances of a component that can be configured differently.

Third, a component implements protocols/interfaces and access other components via their protocols/interfaces.

And most important, the other components that a component depends on are configurable. In java, it would mean injecting one or more component factories into a component.

Given this, any automation of the initialization process will have limited use.

This is why I believe in ultra-light components, where the component close frees up the links between a component and the component it depends on. Now you can have variants where some types of components can only be used/owned by one other component and other types of components need to maintain a reference count to control when they automatically close. But the second case is something I have never needed and is usually handled by separating ordinary components from framework components.

[Brian Platz]

>> This is also something that wouldn't be possible with Mount as this library seems to promote global state.

As a recent switcher from Component to Mount, and without trying to change the thread's topic into a this vs. that -- I'll simply say that I don't believe any of these tools promote global state, it is people who code global state, and that can be with any of these tools... or likewise avoided with any of these tools.

Some tools (i.e. Component) probably make it more difficult to have global state, but I think it is heavy handed. For projects with a lot of components, I would spend a lot of time backtracking components all feeding into each other to figure out where some var was when working in REPL. I'd also repeatedly deal with errors when adding new components as I didn't set up the dependencies correctly at first... just several interlocking pieces that all need to be coordinated, and I sometimes forget one (or two).

Mount probably makes it a little easier to have global state, but that is up to the developer - I have no more global state than I had before the switch. I find it easier to work in REPL and get access to a var, or conn, etc. when I need to eval something, and I think all these components are primarily there to make the REPL workflow better. Also, I'm out of the business of managing my dependencies, which my challenges might just root from an absent-mindedness that I possess. Once it is in production, the component stuff matters very little anyhow.

All to say that these tools, assuming they provide the feature needs that have been outlined well in this thread, should not make anything 'not possible' and can have as much or as little global state as the developer chooses to code in. I cringe a bit when I repeatedly see that Mount promotes global state, I think that is a falsehood.

-Brian

[Jeroen van Dijk]

Hi Brian,

When looking at the Readme of Mount (I think) I already see global state backed in. 

(defstate ^{:on-reload :noop} 
          mem-db :start (connect config) 
                 :stop (disconnect mem-db))

Do I misunderstand this or do we just disagree on what global state is? What if I want to have several (different) `mem-db` instances, how would that work? 

[Brian Platz]

Jeroen,

Happy to talk more about it on Slack.

No matter what you are `def`-ing something somewhere. For Component I’d `def` a big config map, and I do the same with Mount. My advantage with Mount in the REPL is that I can have local vars for ‘components’ that are easy to reference (but still driven from the main config map). In Component I’d accomplish this by outputting what I need to a var, usually in the ‘user’ namespace that I could get a handle to.

As for several different `mem-db`s, if you mean swapping them out for dev/testing, that is explained here: https://github.com/tolitius/mount#swapping-alternate-implementations

-Brian

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/hYFEpJ6w80k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Francis Avila]

I believe by "several different mem-dbs" he means different instances, not implementations, i.e. they differ by name and (maybe) config, but share an implementation.

There are two ways this can happen: two system graphs in the same clojure runtime, or one system graph that uses the same service-type twice.

In Component, the first case is handled by making a separate Component map; the second by adding a new key to the map.

In mount, the first case is AFAIKT not possible (maybe some swap-with-states trickery?), and the second by adding a new defstate var somewhere.

In general, what Component does with system maps, map keys and records mount does with namespaces. The upside is you don't have to be as manual and explicit about instances, names, dependencies, instantiation, etc (just use normal namespace machinery and accept initialization as a side-effect of require). The downside is that namespaces are singletons, so your system must be a singleton too.

[Derek Troy-West]

I've quite enjoyed reading the various approaches to using Component (or not as the case may be), so much so that I've added my own two pence:

http://derek.troywest.com/articles/how-i-use-component