[ANN] The Kiln, an Evaulation Strategy for Insanely Complex Functions

[Jeffrey Straszheim]

The Kiln is an evaluation strategy for insanely complex functions. It was designed based on two things: my experience with managing several large, complex, ever-changing web applications in Clojure, and my experience in dataflow approaches to modelling.

I have released version 1.0.0 on Clojars. Also, there is quite a bit of documentation and explanation on the project Github page, including a full sample application presented in a somewhat “literate” style.

Please take a look.

http://github.com/straszheimjeffrey/The-Kiln

[Paul deGrandis]

Can you give a better example of insanely complex functions?

I actually think that FP is ideal at modeling, representing, and
managing data-flow, stream-based, or request/response data.
Kiln seems to encourage a sort of imperative style to these
operations, where associative data (like hashmaps) would normally be
used.
What advantages and tradeoffs does Kiln offer? What problems does it
solve for you where you've deployed it?
Is Kiln's goal to create a uniform interface to a finite state machine
-esque concept?

Thanks!
Paul

[Jeffrey Straszheim]

I detail what I mean about "insanely complex" here:

http://github.com/straszheimjeffrey/The-Kiln/wiki/Why

And I would reject any description of the Kiln as "imperative", as such.
It does allow for side effects, but that is a matter of design and taste.
You could use it without side effects.

Ultimately, it is a dataflow model, albeit one where each cell
is only be computed once. This makes it the opposite of a
"cells-alike" library, where you want to model statefull cells
changing. The Kiln is for when your cells do not change, at
least over the lifetime of a particular kiln.

During a webapp request, the `current-user` is a fixed value, as
is `request-uri` and `response-main-body`, etc. . I find that managing
the dataflow between this stuff is an endless headache.

And no, it is not a FSM. FSM's model things that change. Kilns
model things that do not (over the lifetime of a single kiln).

[Andrew]

Cool... Do you use kilns at Akamai, and to what extent?

Another question: you set up coals and clays and eventually kilns are fired. When you're setting up the coals and clays in code, you're telling the system about dependencies. Are these dependencies laid out explicitly enough to be always unambiguous-in-execution? 

(I'm comparing this to an architecture in the object-oriented world where we ditched static singletons since static things in Java are initialized at load time and we had no control over the sequence. In our architecture, we had explicit metadata lists for depends-upon and provides-interface and everything was registered with a master collection which would construct a valid sequence for set-up and tear-down according to those metadata lists. So the ambiguity was eliminated with the use of those lists. And the issue of passing around more and more arguments was addressed since now you pass around the single master collection. And each context could have its own master collection... etc.)

What were the other ways of managing complexity that you considered and why did you decide that kilns would fit your needs better?

[cperkins]

I like it. Kiln looks like it is automatically composing the request
handler based mostly on a description of data types (*) provided and
needed. Is that correct, more or less?

It looks very useful. Using Common Lisp (not Clojure yet) I end up
using a lot of macros when handling HTTP requests to handle
boilerplate minutiae but that strategy doesn't always work cleanly
because most of the requests are the almost the same but slightly
different. So either I write macros with arguments to configure the
output of the macro or end up making multiple macros, or resort to
inserting boilerplate by hand.

[Jeffrey Straszheim]

On Tuesday, May 8, 2012 11:18:42 AM UTC-4, Andrew wrote:

Cool... Do you use kilns at Akamai, and to what extent?

Another question: you set up coals and clays and eventually kilns are fired. When you're setting up the coals and clays in code, you're telling the system about dependencies. Are these dependencies laid out explicitly enough to be always unambiguous-in-execution? 

(I'm comparing this to an architecture in the object-oriented world where we ditched static singletons since static things in Java are initialized at load time and we had no control over the sequence. In our architecture, we had explicit metadata lists for depends-upon and provides-interface and everything was registered with a master collection which would construct a valid sequence for set-up and tear-down according to those metadata lists. So the ambiguity was eliminated with the use of those lists. And the issue of passing around more and more arguments was addressed since now you pass around the single master collection. And each context could have its own master collection... etc.)

What were the other ways of managing complexity that you considered and why did you decide that kilns would fit your needs better?

I cannot be very specific on what is happening at my work. But I will say the Kiln resulted from my experience here.

The dependency graph is implicit in your clay arrangement. However, I do no static analysis of the graph. If you create a cycle, it will fail at runtime. The stack trace should be adequate to see what went wrong.

It was a tradeoff. To make dependency analysis automatic would require the call graph be explicit at compile time. I decided that the ability to pass around coals/clays as first class objects, and to evaluate them or not as needed, was more important. Also, they obey all the normal Clojure rules on namespacing and lexical scope. To detect cycles and such would require global knowledge of all namespaces plus control-flow analysis. That would be a very different kind of library.

However, in practice I expect it to work well. Since clays name conceptual values, their dependencies should easily map into particular categories. That is, a developer may say, “This is part of the dispatch system, so I cannot count on data from the search result system, as that cannot fire until I have resolved dispatch.”

From what you say, I am guessing your Java system was able to do entirely static analysis of the call graph.

 

[Jeffrey Straszheim]

Well, I’m not sure what you mean. It does nothing specific with the “data types” as such, so I would say, no, that isn’t it.

[Marc Limotte]

Hi Jeff,

What do you think about a Map interface for this?

I recently implemented something similar in a project of mine, which I called an 'evaluating-map'.  It's not a Web project, but the pattern is a general one.  In my case, a DSL for specifying a job to run.  I want the DSL writer to have access to a lot of data/logic which can come from a lot of different sources (a "big ball of mud" to use your term).

Like you, the mud-ball could contain values or functions.  These functions can have references to other values in the "ball of mud".  I expanded this to include interpolated strings (e.g. "foo is ${foo}") and collections of values/Strings/functions which are handled recursively.  My code doesn't do anything to manage state, although users are encouraged to provide memoized functions and a helper is provided to assist with this.

Here's an example comparable to your example from the Kiln project.

(def m 

  {:request "foo"

   :uri #(build-uri (:request %))   ; an anonymous function works, or

   :path (lfn [uri] (.getPath uri)) ; use lfn, a helper that returns a fn

   :dispatch 

     (lfn [path] (condp = path 

                   "/remove-user" :remove-user

                   "/add-user" :add-user

                   "/view-user" :view-user))

   :action! action                  ; assuming action is defn'ed elsewhere

   ... and so on ... })

lfn is the helper that I mentioned-- it pulls it's args as keys from the "ball-of-mud" and returns a memoized fn of those args.

Eventually, you fire it.  Like Kiln, the concept is that you have a bunch of code that sets it up and then at some point you mix in a few seed values and kick it off.  My fire function does a bunch of other stuff, but the relevant part boils down to (-> m (assoc :request req) evaluating-map), which is used like this:

(let [k (-> m (assoc :request req) evaluating-map)

      result (try 

               (:action! k)

               (render-template (:template k) ...other kiln data...)

               ... catch)]

  ; because it's a Map, you can do things like

  (log/debug (select-keys k [:uri :path]))

  result)

I didn't write support for glazes and cleanup.  I think glazes could be done ring-style.  Cleanup requires some extra thought.  Those are nice features of Kiln.

I think Kiln gives you more control over the execution and state, making things like cleanup easy.  What I like about the Map interface, aside from the convenience of being able to use standard collection functions (merge, select-keys, dissoc, etc) is that you can construct the map from many different sources.  I.e. you can merge maps which are constructed dynamically at different points in your flow.  This was important for my use-case, since DSL users are writing code that my core code knows nothing about.  Using the example above, a subsequent user could replace the :uri fn:

(merge m {:uri (lfn [request] (some-other-build-fn request))}) 

This new function would then be the input for the :path function.

My code for this abstraction isn't isolated, but you can see it in context of another project here.

Anyway, I like the project and thanks for sharing it.

Marc 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Paul deGrandis]

Marc,

Nicely done! I've done something similar before as well.

This was basically my point earlier: Dispatch tables (and maps in
general), unification, and closures seem to solve the same problem,
are composable, don't introduce nouns/types/metaphors, are open for
extension and closed for modification, and if crafted well, can be
shipped around to any reader. Using RabbitMQ and Clojure as my data
format, I can easily create loosely coupled, highly cohesive services
that can be replicated, redundant, etc.

I think the project looks interesting, but the benefits, tradeoffs,
and design decisions aren't clear to me.

Paul

> here<https://github.com/TheClimateCorporation/lemur/blob/master/src/main/c...>
> .

>
> Anyway, I like the project and thanks for sharing it.
>
> Marc
>

[Jeffrey Straszheim]

Looks awesome.

I think we're going the same direction. Myself, I wanted clays to be first-class items that can live in Clojure namespaces, and give you all of that. The downside is this: if I particular clay is wrong for some particular evaluation, you're stuck. The clay is the clay is the clay.

There are a couple ways around this. For testing, with-redefs has its normal purpose. You can also use a function unsafe-set-clay!! to force a clay's value. But that's heavy handed. More usefully, clays can be built lexically, as in

 (let [a-clay (clay :value (... stuff ...) :name something)]

  ... stuff ...)

And passed around

(fire kiln clay-with-args a-lexical-clay)

(defclay clay-with-args

   :args [some-clay]

   :value (blah (?? some-clay)))

This would let you build a computational graph on an as-needed basis.

But still, those are clumsy tools: fine for a few edge cases, but if needed often you'd want something built for the task. For your use case, it sounds like your approach is better. For the specific applications that led to my thinking, the entities were pretty well-defined and their computation well-known. Making them top-level named objects seems the right way to go.