[ANN] Jig

[Malcolm Sparks]

A few months ago, Stuart Sierra blogged about the workflow he follows for building Clojure applications.

"One of the great pleasures of working with a dynamic language is being able to build a system while simultaneously interacting with it. "
-- http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded

Since then I've been using this workflow for my own projects, and found it to be amazingly effective.

I've added some extra features, and the result is Jig, which builds on Stuart's work in the following ways :-

• Multiple components can each contribute to the 'system' map
• Components are started in dependency order
• Components are specified and configured in a config (edn) file
• Jig can host 'plain old' Leiningen projects - Jig will even 'reload' them too e.g. if their project.clj dependencies change.
  
• Jig can host multiple projects simultaneously

There's a small but growing list of optional re-usable components that provide extra functionality :-

• Pedestal services support. Jig provides the system map and 'url-for' function in the service context.
  
• Nginx cache purging on reload
• Git pull prior reload
  
• Reload via JMX
• nREPL
• Stencil cache purging
• Firefox remote control support for 'browser refresh on reload'

I know others are working on similar designs. I'd be interested to hear what people think and whether this is useful.

Thanks,

Malcolm

PS: Jig can be found here: https://github.com/juxt/jig

[Manuel Paccagnella]

Looks very interesting, thanks for sharing! BTW, documentation is impressive and quite comprehensive.

On a related note: I've spotted a couple of typos in your README.md so far. Do you accept pull requests for small fixes like these?

[Timothy Washington]

This looks really cool. A few things. 

A) Looks like I have to run nrepl in the jig project directory. I'm finding that emacs also has to be started in the same jig directory. Then all of my code editing has to be referenced from that directory (ie: C-x C-f ../my-project/my-file.clj). If I try to run emacs in my-project, then M-x repl, I get the below error. Is there a way around this? 

... in *Messages* 

Connecting to nREPL on 127.0.0.1:4555...

open-network-stream: make client process failed: connection refused, :name, nrepl, :buffer, *nrepl-connection*, :host, 127.0.0.1, :service, 4555, :nowait, nil 

... in emacs' minibuffer line 

make client process failed: connection refused, :name, nrepl, :buffer, *nrepl-connection*, :host, 127.0.0.1, :service, 4555, :nowait, nil 

B) I'm trying to wire up an external project, but getting a NullPointerException when calling (go). I'm following the usage instructions, and guide for external projects. Is there something obvious that I'm missing? 

~/Projects/jig$ lein repl

nREPL server started on port 33095 on host 127.0.0.1

REPL-y 0.2.1

Clojure 1.5.1

Welcome to Jig!

(go)       -- start the system

(reset)    -- reset the system

(refresh)  -- recover if a reset fails due to a compilation error

(menu)     -- show this menu again

Jig user> (go)

NullPointerException   clojure.core/val (core.clj:1489)

Jig user> (pst *e)

NullPointerException 

        clojure.core/val (core.clj:1489)

        loom.graph/build-graph/build--1546 (graph.clj:411)

        clojure.core.protocols/fn--6037 (protocols.clj:127)

        clojure.core.protocols/fn--6005/G--6000--6014 (protocols.clj:19)

        clojure.core.protocols/seq-reduce (protocols.clj:31)

        clojure.core.protocols/fn--6028 (protocols.clj:48)

        clojure.core.protocols/fn--5979/G--5974--5992 (protocols.clj:13)

        clojure.core/reduce (core.clj:6177)

        loom.graph/build-graph (graph.clj:425)

        clojure.core/apply (core.clj:619)

        loom.graph/digraph (graph.clj:437)

        jig.system/get-digraph (system.clj:87)

nil

fig.1 - shell error output 

{:components

 {:recordo {:jig/component org.recordo.core/Component

               :jig/project "../recordo/project.clj"}}}

fig.2 - jig/config/config.edn

(ns recordo.core

  (:import (jig Lifecycle)))

;; A Jig Component

(deftype Component [config]

  Lifecycle

  (init [_ system] system)

  (start [_ system] system)

  (stop [_ system] system))

fig.3 - recordo/src/recordo/core.clj; ... recordo's project.clj has no mention of "jig"

Cheers - great work. 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Malcolm Sparks]

Hi Tim,

Thanks for the feedback.

A. By default, nrepl-jack-in will invoke lein from the directory of the project that owns the file in the current Emacs buffer. You need to start Jig from its own project directory, not the project directory of your own application. Normally I load up Jig's project.clj into an Emacs buffer, run nrepl-jack-in from there, and then type (go) in the repl buffer that appears. Alternatively you can type 'lein repl' from the Jig repo directory.

B. This is my mistake. An earlier version of jig used :components while the current version uses :jig/components. I forgot to update the sample.config.edn file. This is now fixed on master.

Everything else you've done looks OK to me.

I keep my own working config for stuff I'm currently running in Jig on a branch (juxtweb), treat it as a working example until I have documented this area more fully:  https://github.com/juxt/jig/blob/juxtweb/config/config.clj

Regards,

Malcolm

[Malcolm Sparks]

On Sunday, 13 October 2013 01:21:40 UTC+1, Manuel Paccagnella wrote:

Looks very interesting, thanks for sharing! BTW, documentation is impressive and quite comprehensive.

On a related note: I've spotted a couple of typos in your README.md so far. Do you accept pull requests for small fixes like these?

Yes please!

[Timothy Washington]

Schweet. That did the trick. 

Ok, I can't wait to start using this. If you like, I made a pull request to update the README. A small change just to save you time. 

Cheers 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[zcaudate]

Would it be possible to put up a video of a typical workflow example with pedestal. It's quite difficult for me to piece everything together just by reading the documentation.

Chris

[Timothy Washington]

Also, is there a Google Group for Jig? I'm playing around with it, can see the potential, and already have some questions. Anyways, in the meantime, I'll keep digging. I'm finding jig really useful so far. 

A) As taken from the example, I'm using the Jetty :server component. I'm trying to figure out how to pass in my Compojure route handler (also see) to jetty. This is how its done manually. But obviously, I need to fit that into the jig framework. 

  :server {:jig/component jig.web.server/Component      ;; how do I pass in my Compojure route handler ? 

           :io.pedestal.service.http/port 8000


           :io.pedestal.service.http/type :jetty
           }

B) Same thing with Pedestal. I want to be able to pass in my app's routing table. But I don't quite grok how to do that, from the examples (here and here). 

  :juxtweb/web {:jig/component jig.web.app/Component    ;; how do I pass in my routing table of handlers ? 

                :jig/dependencies [:server]

                :jig/scheme :http


                :jig/hostname "localhost"
                :jig.web/server :server

                }

Thanks 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Malcolm Sparks]

Hi Tim,

Good to hear you're finding Jig useful.

There isn't any explicit support for Ring in Jig yet, it's just Pedestal services right now. The separation of jig.web.server and jig.web.app into 2 components is to allow multiple 'virtual hosts' to share the same Jetty server, if necessary.

It's a third component, usually your own, that actually adds routes to a virtual host. See the ReadmeComponent in examples/docsite/src/examples/docsite/core.clj here. It adds routes in its init phase.

The idea is that other components can be contribute routes in the same way, so you have the option of breaking up a website into modules.

The nice thing about Pedestal services is that hyperlinks can be generated using the url-for functionality it provides. If you use this feature consistently, you can treat the jig.web.app as an anchor on which to hook multiple applications and configure the URL tree accordingly. I'm intrigued by this url-for feature and wanted to experiment with it with Jig.

Since Compojure doesn't have this automatic ability to create links from handler vars, it's likely you'll want to set it up manually. Here's a suggestion for how to do this:

1. Create a Ring server component.

2. In the init phase, add an empty vector called :routes. Other components can depend on this one, and in their init phase, they concat their routes to this vector.

3. Then, in its start phase, the Ring server component composes the routes together (using Compojure's routes function) and creates and starts the server, placing the server in the system map so that it can be found and stopped in its stop phase.

Remember that you don't need to use any reloading tricks described by Ring, such as passing a var to run-jetty rather than the handler itself. Nor do you need :reload and any file status detection.

What I really like about Stuart's workflow is that it allows you to do away with these 'dev versus prod' modes, you just code everything in 'prod' mode (i.e. there is only one mode) and let the reset handle the dev reload. This makes code simpler and ensures you are developing and testing what the users get, not a development 'version' that approximates to it.

Regards,

Malcolm

[Malcolm Sparks]

Hi Chris, yes I will try to do something like that very soon.

[Timothy Washington]

Hey Malcolm, 

So since you guys just have Pedestal set up at the moment, I've started there. 

My Repos 

You can see my project jig here. It connects to common, compojure (not yet configured), and pedestal-service component. I'm trying to get the Pedestal portions working. Now, I haven't yet been able to stitch together routes using the Pedestal ":juxtweb/service" component you have. The ":server" component runs fine, when executing (go). But the handlers don't seem to be attached. Do I need a separate ":jig/projects" entry in my config? 

{

 :jig/components

 {

  ;; ===>

  :server {:jig/component jig.web.server/Component

           :io.pedestal.service.http/port 8000

           :io.pedestal.service.http/type :jetty}

  :juxtweb/web {:jig/component jig.web.app/Component

                :jig/dependencies [:server]

                :jig/scheme :http

                :jig/hostname "localhost"

                :jig.web/server :server}

  ;; ===>

  :stefon-webui-common {:jig/component stefon-webui-common.core/Component

                        :jig/project "../stefon-webui-common/project.clj"}

  :stefon-compojure {:jig/component stefon-compojure.core/Component

                     :jig/project "../stefon-compojure/project.clj"

                     :jig/dependencies [:server

                                        :stefon-webui-common]}

  :stefon-pedestal {:jig/component stefon-pedestal-service.service/Component

                    :jig/project "../stefon-pedestal-service/project.clj"

                    :jig/dependencies [:juxtweb/web

                                       :stefon-webui-common]

                    :jig.web/app-name :juxtweb/web}

  }}

Juxt's Cloned Repos 

So I tried cloning your example repos to see working example code. But there's a few things that went awry. 

A) I had to rename the cloned repo for the referencing in  "jig/config/config.clj" to work (:jig/project "../accounting/project.clj"). 

~/Projects/trial/JIG$ ls -la

   accounting/   ;; had to `mv juxt-accounting accounting` 

   jig/

   juxtweb/

B) Also, you've hard-coded an accounts db file to your hard drive

java.io.FileNotFoundException: /home/malcolm/Dropbox.private/JUXT/accounts.edn

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Malcolm Sparks]

Hi Tim,

I've just pushed Jig 1.2.0 to https://github.com/juxt/jig.

This release adds support for Ring and Compojure, which many people have asked for, including proper examples of using both Ring/Compojure and Pedestal services. There are numerous other components included (Git pull, JMX, nginx) to support an automated continuous deployment process.

As you pointed out, there were missing details in my own work branches, so the packaged examples should hopefully make things clearer.

Thanks for the feedback on using Jig, it is much appreciated, and please let me know if you still need some help with the specifics.

Regards,

Malcolm

[Timothy Washington]

This is great. I tried digging into the code and saw the repo changing. So I'm glad I didn't get too far down a path. I'll definitely have a look in the next few days and have feedback for you. 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Chris Zheng]

Thanks Malcom,

Even something really simple like showing your current project setup would be really good for me to get going.

--

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/E0BdR_AksiA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Timothy Washington]

Hey Malcolm, 

A) Ok, so just playing around with jig some more. I'm looking to get working Compojure and Pedestal examples. 

B) I started with a raw `git clone g...@github.com:juxt/jig.git`. Then in that directory, I ran the commands in blue. 

C) Now, I run my environment in a Virtual Machine. So when using the browser to try and go to each of the url end points, I got an HTTP 404, "Not Found" error for all of these URLs. Ie, Compojure and Pedestal were running, giving me HTTP responses. They just weren't giving me the advertised endpoints, only 404s. This corresponded errors I was experiencing when I tried to stitch in one of my simple compojure handlers in a separate jig. There's probably some small config I'm missing, before this becomes reeeally useful. Can't wait :) 

• http://172.16.210.128:8091/server
  
• http://172.16.210.128:8001/readme
  
• http://172.16.210.128:8001

$ lein repl

nREPL server started on port 59398 on host 127.0.0.1

REPL-y 0.2.1

Clojure 1.5.1

Welcome to Jig!

(go)       -- start the system

(reset)    -- reset the system

(refresh)  -- recover if a reset fails due to a compilation error

(menu)     -- show this menu again

Jig user> (go)

INFO  jig.system - Starting the system

INFO  jig.system - Starting component ':test-ring-handler'

INFO  jig.system - Starting component ':compojure'

INFO  jig.system - Starting component ':jetty'

INFO  org.eclipse.jetty.server.Server - jetty-8.1.9.v20130131

INFO  o.e.jetty.server.AbstractConnector - Started SelectChann...@0.0.0.0:8091

INFO  jig.system - Starting component ':console/server'

INFO  org.eclipse.jetty.server.Server - jetty-8.1.9.v20130131

INFO  o.e.jetty.server.AbstractConnector - Started SelectChann...@0.0.0.0:8001

INFO  jig.system - Starting component ':console/web'

INFO  jig.system - Starting component ':console/readme'

:ready

Jig user> INFO  i.p.s.http.impl.servlet-interceptor - {:line 362, :in :interceptor-service-fn, :context {:servlet #<FnServlet io.pedestal.service.http.servlet.FnServlet@606acc48>, :servlet-config #<Config org.eclipse.jetty.servlet.ServletHolder$Config@19017f49>, :servlet-response #<Response HTTP/1.1 200 

>, :servlet-request #<Request (GET /)@1536124838 org.eclipse.jetty.server.Request@5b8f67a6>}}

INFO  io.pedestal.service.http - {:line 58, :msg "GET /"}

INFO  i.p.s.http.impl.servlet-interceptor - {:line 362, :in :interceptor-service-fn, :context {:servlet #<FnServlet io.pedestal.service.http.servlet.FnServlet@606acc48>, :servlet-config #<Config org.eclipse.jetty.servlet.ServletHolder$Config@19017f49>, :servlet-response #<Response HTTP/1.1 200 

>, :servlet-request #<Request (GET /web)@1536124838 org.eclipse.jetty.server.Request@5b8f67a6>}}

INFO  io.pedestal.service.http - {:line 58, :msg "GET /web"}

INFO  i.p.s.http.impl.servlet-interceptor - {:line 362, :in :interceptor-service-fn, :context {:servlet #<FnServlet io.pedestal.service.http.servlet.FnServlet@606acc48>, :servlet-config #<Config org.eclipse.jetty.servlet.ServletHolder$Config@19017f49>, :servlet-response #<Response HTTP/1.1 200 

>, :servlet-request #<Request (GET /readme)@1536124838 org.eclipse.jetty.server.Request@5b8f67a6>}}

INFO  io.pedestal.service.http - {:line 58, :msg "GET /readme"}

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Timothy Washington]

Ok, I actually got the compojure example working. I just had to remove the config/console.edn and config/default.edn files in my jig. They must be disrupting the config that I put in. So that's my only feedback so far. 

Ok, this is looking really good. Great work :)  

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Timothy Washington]

Ok, some more feedback for Jig. 

A) Testing - Let's say I have a multi-project jig, with dependencies across projects. There doesn't seem to be a way to run tests (midje :autotest) for a specific project. I tried creating a Midje component (see https://www.refheap.com/20442). But when I i) put this into config.edn, and ii) thread through my local project component, iii) this only prints out the classpath directories under the jig project. I'll want to be in the Classpath context of the project that's being passed in. That way, I can :autotest for that project. Or perhaps there's another way to run and autorun tests for a particular project. I noticed that Juxt-Accounting has a test suite. 

B) Ok, so there's just that and removing the config/console.edn and config/default.edn files when writing your jig. 

C) And a raw git clone g...@github.com:juxt/jig.git, then lein repl, then  (go), doesn't give working URLs ( http://<>:8091/readme , http://<>:8001/server, etc )

Let me know if you want me to log these as bugs, feature requests, etc. Very excited about the kind of leverage that this project can yield. 

Loving this tool. Thanks. 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

[Malcolm Sparks]

Hi Tim,

Thanks for the great feedback, it's really useful.

I've tried to keep the config so that it works out-of-the-box for examples, but is possible to point at other configurations without mutating the Jig repo itself. But I see there's more refinements needed.

If you look in jig/src/user.clj you'll see a (config) function - that tries a number of locations in order to find a config file. For example, it looks for a .jig/config.clj file before looking in jig/config/config.clj.

Regards,

Malcolm

[Malcolm Sparks]

Hi Tim,

The tests in JUXT accounting are using clojure.core.test. I'm fairly sure Midje's :autotest feature does something dynamic to determine the tests to run and that may not work with Jig's classloading approach. For example, if something uses (System/getProperty "java.class.path") it will just get the Jig source directories because, for external projects, Jig must load these using a child class-loader. I need to spend some time with Midje to work out what it's doing.

Having multiple lein projects loaded in the same JVM, and integrated with each other, is not a common Clojure mode of usage today. However, the Immutant team (and others) have done a lot of the groundwork and I think the various caveats I've listed on Jig's README.md about 'external projects' are going to be ironed out over time as these issues become better understood. I expect that Brian never saw this as a use-case.

In the absence of a mailing list right now, please log any issues you see as GitHub issues and I'll do my best to fix them.

Thanks again for sending in this really useful feedback.

Regards,

Malcolm

[Timothy Washington]

Too right. 

Yes, wrt the multi-project / classpath thing, running isolated test for a particular project is only one aspect. I also have an eye to running a i) browser-connected repl and ii) debugger for a particular project. So those things, along with iii) running tests, make very high, the attractiveness of having that isolation. 

I'll try to put those in github issues. And also pick at the problem myself when I get some cycles. I think it would add a powerful feature to the tool. Anyways... :) 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Malcolm Sparks]

Hi Tim,

It's interesting you're thinking about browser-connected repls. Today I added clojurescript support to Jig, it's pushed to master but I haven't documented the config options yet, but below is an example. Nothing fancy like crossovers yet, it's basically a thin wrapper over ClojureScript's compiler, the config options map directly to the options given to the compiler.

  :cljs-builder
  {:jig/component jig.cljs/Builder
   :jig/project "../my-proj/project.clj"
   :source-path "../my-proj/src-cljs"
   :output-dir "../my-proj/target/js"
   :output-to "../my-proj/target/js/main.js"
   :source-map "../my-proj/target/js/main.js.map"
   :optimizations :simple
   :pretty-print true
;;   :clean-build true
   }

Another built-in component can figure out, given the dependencies, what files to serve and on which server to serve it from.

  :cljs-server
  {:jig/component jig.cljs/FileServer
   :jig/dependencies [:cljs-builder :web]
   :jig.web/context "/js"
   }

which depends on some Ring or Pedestal routing and Jetty or other webserver, but you're likely to have that configured already :-

  :server
  {:jig/component jig.web.server/Component
   :io.pedestal.service.http/port 8000
   :io.pedestal.service.http/type :jetty
   }

  :web

  {:jig/component jig.web.app/Component
   :jig/dependencies [:server]
   :jig/scheme :http
   :jig/hostname "localhost"
   :jig.web/server :server
   }

That's it really. I think Jig makes an ideal platform for building and serving clojurescript, in comparison to Leiningen plugins. Firstly, there's no JVM start-up cost, the point of Jig is to keep the JVM running all the time, so there's no need for the once/auto service that lein-cljsbuild offers. Secondly, serving the resulting Javascript files is straight forward, and built in, so no 'lein ring server' requirement either.. With Leiningen, you either have to build routes to the javascript into your own app, or run lein ring server, but then Leiningen doesn't make it easy to run multiple services concurrently. Thirdly, there's no file-system polling, and services that depend on the clojurescript compilation can be started as soon as the compilation is complete.

One thing that lein-cljsbuild does really well is multiple build configs. I've decided to use a single build configuration, to keep implementation easier and a direct mapping to the clojurescript compiler, but of course you can add different builds by just adding (differently configured) components to the Jig config. This would benefit from being able to initialize and start components in parallel, where possible. Unlike cljsbuild, Jig components are started serially, in reverse dependency order. Does anyone know of existing algorithms that can walk a dependency tree in parallel?

Feel free to take Jig in different directions though, I'm just letting you know my current thoughts. The single REPL to multiple projects idea might have to be rethought - it seems the prevailing wind is in the direction of multiple REPL connections per editor/IDE.

[julius]

+1

[julius]

Another question, using jig,  the connection/db/cache/storage will be everywhere in our code as a parameter of functions, is it flexible? ,currently I prefer to managing those side effect at one place but will not spread out to our other core functions.

Thanks

[Timothy Washington]

Umm I can't think of parallel dependency tree walking algos, off the top of my head. Sorry :/ 

But niiice. Yeah, we're definitely thinking along the same lines. But you're way ahead of me. Now just to be able to isolate that browser repl on a per project basis. 

Sweet 

Tim Washington 

Interruptsoftware.ca / Bkeeping.com 

On Wed, Nov 6, 2013 at 8:46 PM, Malcolm Sparks <mal...@juxt.pro> wrote:

Hi Tim,

[Cedric Greevey]

If you are just visiting every node as a tree, and you have core.async, then there's

(let [c (chan)]

  (go

    (doseq [n (tree-seq ... (depends on tree implementation) ...)]

      (>! c n)))

  (dotimes [_ (.availableProcessors (Runtime/getRuntime))]

    (go

      (loop []

        (when-let [n (<! c)]

          (expensive-visitation n)

          (recur))))))

unless there's some trickiness with core.async that I've not "gotten" here.

Of course, if expensive-visitation returns something instead of having side effects, you'll need to accumulate some sort of a result in each loop, and combine these afterward (the dotimes likely becoming a for).

If the visitation is really expensive, then converting the output of tree-seq into a vector and using reducers or pmap might also parallelize things significantly.

If you need to do more than just visit each node -- if the tree structure around the node and not just some value in the node needs to be considered -- then it gets complicated. Something that traverses the tree, but spawns new threads at each branching (up to some limit perhaps), seems indicated.

--

[Malcolm Sparks]

This is a good question, but not so much one for Jig as for Stuart's workflow pattern on which Jig's based.

Generally, I write my Clojure bottom-up, so that functions that do work against database connections and the like, take these as parameters, and not the whole system map. This leads to the situation where callers to these functions are doing stuff to extract the necessary values from the system map in order to pass them as parameters. Also, some functions have to take a lot of parameters. So at certain times I feel it's often easier to pass down the whole system map. This seems in violation of http://en.wikipedia.org/wiki/Principle_of_least_privilege - however, it's much better to pass state in to a function as a parameter, than have that function pull in state from vars that are scattered around various name-spaces. This also has a benefit to testing - it's possible to construct mock system map to test functions.

On the plus side, there is some precedent here. Ring handlers take large request structures, Liberator and Pedestal handlers take similar large 'context' parameters. Datomic's query takes a whole database as a parameter! There is good support in Clojure for handling larger structures, such as clojure.walk, de-structuring, and so on. This would be a good time to mention Strucjure https://github.com/jamii/strucjure on which Jamie Brandon gave a great talk earlier this week in London. A common pattern in Strucjure is to use a tree walk and match on parts of the structure to drive processing.

[Malcolm Sparks]

That's a nice use of core.async  :)

The key requirement, however, is that a component's dependencies must all initialize before a component can start initializing. That reduces parallelism but not entirely. A component's dependencies can all be initiated in parallel, so long as there aren't any dependencies between each other. I'm sure there must be existing barrier sync algorithms out there that we could use. http://en.wikipedia.org/wiki/Barrier_%28computer_science%29 

[Cedric Greevey]

Ah, for that you need to "layer" your DAG (directed acyclic graph -- if A and B both depend on C, and are dependencies of D, you don't have a tree). Start with S an empty set and n = 0. Iteratively let Sn be the items with no dependencies not in S, S become S U Sn, and n = n + 1 until no items are not yet in any of the Si. Now initialize, perhaps in parallel, all of the S0; then all of the S1; then all of the S2 ...

[Ivan Kozik]

On Sat, Nov 2, 2013 at 11:05 PM, Timothy Washington <twas...@gmail.com> wrote:
> C) Now, I run my environment in a Virtual Machine. So when using the browser
> to try and go to each of the url end points, I got an HTTP 404, "Not Found"
> error for all of these URLs. Ie, Compojure and Pedestal were running, giving
> me HTTP responses. They just weren't giving me the advertised endpoints,
> only 404s. This corresponded errors I was experiencing when I tried to
> stitch in one of my simple compojure handlers in a separate jig. There's
> probably some small config I'm missing, before this becomes reeeally useful.
> Can't wait :)

I ran into this as well - it turns out that if you have a
:jig/hostname in a jig.web.app component, e.g.

:jig/hostname "localhost"

you can only hit that web component with that particular hostname -
another hostname or an IP results in "Not Found". (A typical virtual
host setup, but a little unexpected in a development environment.)

Ivan

[Joachim De Beule]

Very impressive work! 

I was thinking of letting components not only implement the Lifecycle protocol but also a protocol that defines the functionality of the component. To give a simple example, there could be a jdbc component that establishes a connection on init etc., and which also implements a JDBCProtocol, e.g. functionalities such as provided by clojure.java.jdbc/query etc.  Does that make sense?

I ask because I didn't see any of the extensions or examples done in this way, and because there doesn't seem to be much code that supports fetching and manipulating the system map which seems to speak against this approach?

Op vrijdag 11 oktober 2013 18:23:41 UTC+2 schreef Malcolm Sparks:

[Joachim De Beule]

(follow up)

I just realized that another approach would be to hold the jdbc connection type implementing the JDBCProtocol in system under [:jdbc-component :db] or something, and then call the clojure.java.jdbc/query like functions on that. Anyway, I would be very happy to hear your comments on this!

  

[Joachim De Beule]

Op maandag 21 oktober 2013 02:41:43 UTC+2 schreef frye:

Also, is there a Google Group for Jig? I'm playing around with it, can see the potential, and already have some questions. 

+1 for the google group ..

[Malcolm Sparks]

Hi Joachim

The component lifecycle protocol is used to group together the init, start and stop functions tied to ephemeral services common to many Java server-side applications. I'm not sure it makes sense for components to satisfy other protocols that are functional in nature, as this invite coupling of concerns. The intention is that component functions inject/reject state into/from the system map. Your suggestion of retaining a JDBC connection under [:jdbc-component :db] is exactly what Jig intends. Clojure applications should, as far as possible, avoid state. But when applications need to retain state, they should do so in a system map that is continually subject to repetitive creation and destruction, if only to avoid the detrimental effect that stale state can have on the speedy and iterative development of applications.

I have toyed with the idea of adding core.async protocols and provides/requires metadata to components, but none of my approaches have felt satisfactory. I think it's better to keep components as lightweight as possible, letting them setup and teardown state within the system map, and with that as their single responsibility. (However, if you have alternative suggestions, do let me know)

Regards,

Malcolm

On 22 January 2014 22:22, Joachim De Beule <joachim....@gmail.com> wrote:

(follow up)

I just realized that another approach would be to hold the jdbc connection type implementing the JDBCProtocol in system under [:jdbc-component :db] or something, and then call the clojure.java.jdbc/query like functions on that. Anyway, I would be very happy to hear your comments on this!

  

--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/E0BdR_AksiA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Joachim De Beule]

Hi Malcolm,

Thank you for your very helpful reply! 

Indeed, I've also been thinking about "database components", "elasticsearch components", "memcache components" and so on that all are async/channels (or something) but failed to find the right abstraction so far, so probably it is best to separate concerns (if that is what you mean?)  

I have a follow up question if you don't mind. Suppose I want to define a component that starts a thread and regularly checks a resource. If the resource changes, this has repercussions for other (dependent) components. How would you do that in jig? 

[Joachim De Beule]

Hi Malcolm,

I have a follow up question if you don't mind. Suppose I want to define a component that starts a thread and regularly checks a resource. If the resource changes, this has repercussions for other (dependent) components. How would you do that in jig? 

Maybe this question is too broad, so let me ask some more specific questions. All components should get/set their state from/in the system map, right? Now the system map is a var defined in the user namespace. Does this mean then that the resource-checking thread should alter the system var?  And then I guess it should also call user/reset? Or am I still missing something here?

BTW, not that I want to press you, but do you have any idea when you will be finishing up version 2? I ask because it seems that the github code currently is kind of in-between versions...

Thanks again,

Joachim.

[Malcolm Sparks]

Hi Joachim,

I suggest you create a component, C, that assocs a channel in the system map.

Now create a component, A that depends on C. A should start and stop a thread in the usual start/stop Lifecycle. The component should get the channel from the system map. The thread should check the resource, and if it changes, put messages on the channel.

Components that are dependent on changes to the resource should also depend on C, start go blocks are read from the channel it provides. If you need multiple consumers, a pub/sub mechanism, all that stuff is provided by the core.async API. If you really need to automatically cause a reset, you can do that too - see the API in jig.reset, but it's rarely necessary. I don't think it's what you want to do in this case.

Jig Version 2 is in RC status. This is mostly because the documentation (README) needs to catch up. As soon as I've fully updated the docs I'll release the final 2.0.0.

Regards,

Malcolm

--

[Joachim De Beule]

Thanks so much for your suggestions Malcolm, very helpful and illuminating!

Joachim.

[Joachim De Beule]

Dear Malcolm.

I'm still having trouble seeing the intended flow, and I have the feeling I'm missing something. 

My first question is why I would create two separate components C and A, instead of just having one component A that both installs a channel in the system map and starts a thread for pushing updates to it when the external resource changes? 

More importantly, it's not clear to me what should happen with the updates that are sent over the channel. To make things concrete let's say the resource is a configuration file specifying how to access an external database (url, login and credentials, ...). In order to access the database, I create a database component D providing public functions (fetch-data ...) and (store-data ...) etc. These functions need access to an up-to-date db-spec or connection, which in turn depends on the config values sent over the configuration update channel. 

Now I see three possibilities:

1) Component D subscribes to the config update channel and keeps a private state containing an up-to-date db-spec which in turn is accessed by the public fetch and store functions

I don't think this is the way to go because I thought the whole purpose was to keep all state in a single place (i.e. "system")? This brings me to possibility 2:

2) Component D, upon receiving a config update, updates the system global var root, and the fetch and store functions in turn access the global system to get an up-to-date db-spec

 

I thought this was the way to go, but I'm confused since the system is defined in the user namespace which is not supposed to exist in production?

3) Component D does not subscribe to the config update channel. Instead components that want to access the database should themselves subscribe to the config update channel and provide the store and fetch functions with an up-to-date db-spec.  

But then I feel that this simply pushes the problem one level up and that the db component does sufficiently encapsulate all the internals of accessing the database and maintaining an up-to-date connection etc.?

[Joachim De Beule]

update: I made mistake at the end of my previous post: I mean "that the db component does NOT sufficiently encapsulate the internals" of course...

[Malcolm Sparks]

Hi Joachim,

What you're describing is a dynamic system map, whereas the current pattern as described by Stuart in his classic blog post http://thinkrelevance.com/blog/2013/06/04/clojure-workflow-reloaded describes a static system map which is established at the beginning of a reset cycle and remains. After all, it's a var, not an atom. The reset pattern is intended as a development aid, with the bonus of providing some architectural hygiene that comes in having system state located in a single map (rather than as vars spread over many namespaces). There is no intention to reset a production system.

However, there's no reason why you can't put an atom or ref in the system map, have that atom or ref updated every time the config changes, and connect everything together with core.async. Parts of your program that need up-to-date configuration details can deref the atom or ref, just in time, or watch the atom or ref in order to trigger database re-connections.

You definitely shouldn't be altering the global var root which contains the system, as suggested in 2. You shouldn't access user/system in your program code (although it's a useful thing to access in an exploratory REPL). Instead, a component should pass the system it receives in the start phase as an argument when building structures, such as Ring middle-ware chains, connection managers, thread pools, etc.

Regards,

Malcolm

On 11 March 2014 15:17, Joachim De Beule <joachim....@gmail.com> wrote:

update: I made mistake at the end of my previous post: I mean "that the db component does NOT sufficiently encapsulate the internals" of course...

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/E0BdR_AksiA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Joachim De Beule]

Got it! :-) 

One more thing. While developing, I often have to deal with large amounts of data that take quite a while to load. However, the data is lost and so has to be reloaded after every reset, which is a bit annoying. Is there a way to prevent this? 

[Malcolm Sparks]

It's a nice idea to load your schema and data on every reset, if you can.

But if you need to, place you data anywhere under the :safe key in the system map and it will survive a reset.

On 18 Mar 2014 15:57, "Joachim De Beule" <joachim....@gmail.com> wrote:

Got it! :-) 

One more thing. While developing, I often have to deal with large amounts of data that take quite a while to load. However, the data is lost and so has to be reloaded after every reset, which is a bit annoying. Is there a way to prevent this? 

--

[Joachim De Beule]

Hi Malcolm, me again :-)

I've been using (and loving) jig now for quite some time and have reached the point where I want to start using my code in a production environment. There are several things that need to get fixed though, most importantly:

1) Testing. Have you by any chance thought more about what is the best approach? The main problem still seems to be that it is not possible to test components separately (at least not if they depend on other components). I'dd like to help solve this problem, but it would be good if you could provide some hints on how to best proceed.

2) Daemonizing. So far I've tried with lein-daemon, Tanuki's Java Service Wrapper (see http://techwhizbang.com/2012/02/clojure-applications-as-daemons/) and jsvc (see http://www.rkn.io/2014/02/06/clojure-cookbook-daemons/). None of them work out of the box, presumably because the jig loader strategy relies on relative instead of absolute paths. Again, I'd love to help solve this but could use some hints. 

Thanks!

[Joachim De Beule]

Just a small follow up, in the mean time I did manage to get the code running as daemon using Tanuki's service wrapper by setting the wrapper.working.dir to the main project's root.