concat component?

1 view
Skip to first unread message

Serge Wroclawski

unread,
Apr 29, 2009, 11:36:23 PM4/29/09
to Clojure Study Group Washington DC
I thought I'd start my serious investigation of writing components by
looking at the existing one and the concat one seems like the
canonical answer. Here it is as of last checkout:

(def-component {:name "concatenate"
:category "Transform"
:description "Concats record sequences"
:output-type "Record Sequence"
:args {"inputs" {:doc "The record sequences to concatnate"
:type "Record Sequence"
:min-required 1}}}
(fn [pipe pipe-args args]
((fn cat-inputs [inputs]
(concat
(lazy-seq
(if (seq inputs)
(concat (first inputs) (cat-inputs (rest inputs)))
'())))
(args "inputs"))))


All the components should assume that they're taking sequences as
inputs. If someone throws a non-sequence at a component, they're
taking their life into their own hands (and an exception should be
thrown).

And concat takes as many arguments as you can throw at it and returns
a lazy-seq. Such as in:

(take 2 (concat '({:a 1 :b 2}) '({:a 2 :b 3}) '({:a 3 :b 4})))

So why does this function construct a lazy-seq recursively?

Why doesn't this do the same job?

(def-component {:name "concatenate"
:category "Transform"
:description "Concats record sequences"
:output-type "Record Sequence"
:args {"inputs" {:doc "The record sequences to concatnate"
:type "Record Sequence"
:min-required 1}}}
(fn [pipe pipe-args args]
((fn cat-inputs [inputs]
(apply concat args)))))

My question is do I:

a) Misunderstand concat?
b) Misunderstand the def-component macro?
c) Misunderstand something else?

- Serge

Luke VanderHart

unread,
May 4, 2009, 4:54:09 PM5/4/09
to Clojure Study Group Washington DC
Um... you're right. The version you cite is actually semantically
identical and obviously a lot cleaner.

Somehow I was having a brain fart and didn't realize concat was
already lazy. Doh. I think I'd been working with so many non-lazy
sequence functions right before that I wrapped it in a lazy-seq by
default, which is very unnecessary.

But if it WASN'T lazy, that's how you'd do it ;)

Sorry about that.

-Luke

Serge Wroclawski

unread,
May 5, 2009, 7:15:12 AM5/5/09
to clojure-...@googlegroups.com
On Mon, May 4, 2009 at 4:54 PM, Luke VanderHart
<luke.va...@gmail.com> wrote:
>
> Um... you're right. The version you cite is actually semantically
> identical and obviously a lot cleaner.
>
> Somehow I was having a brain fart and didn't realize concat was
> already lazy.

I was thinking that maybe your def-component macro did something and
un-lazyied the structures.

I know we're thinking of rewriting everything from scratch, but I'd
still like to have you explain the macro. It hurt my head.

- Serge

Luke VanderHart

unread,
May 5, 2009, 10:49:43 AM5/5/09
to Clojure Study Group Washington DC
Sure. It's not too bad, as far as logic flow goes- just a lot of crazy
symbols and macro-language.

I'll refer to thinks by line number on the following link, by the way.
The macro definition starts on line 73.

http://github.com/levand/flowjure/blob/b10b4efd59d08731e056c045e7765f61cc01d149/src/flowjure/engine.clj

The macro takes two parameters. The first is simply a map which will
be attached as metadata. The second is a form which returns a function
(most commonly, a function definition).

Line 81 starts the body of the macro with a backtick, and opens with a
defn. The macro as a whole, then, expands to a "defn" expression. The
fn it defines is actually what is called during pipe creation.

Note that on line 81, the actual name of the function is created by
taking the :name field of component-cfg, creating a symbol from it,
and appending metadata to it. So the ~ expression in line 80 simply
resolves to a symbol with attached metadata.

Line 82 adds the "description" as the "documentation" line in a normal
defn statement.

Line 83 declares that the function will take one argument. We add a #
to invoke the auto-gensym and prevent our parameter name from
capturing any external variable.

The "let" statement on line 84 binds "action-fun#" to the function
definition that was passed in to the macro. Note that we expand it -
this means that we can then _call_ "action-fun#" within our "let".

85-86 just define certain local bindings from the vars *pipe* and
*pipe-args* for convenience. *pipe* and *pipe-args* are clojure Vars,
which obey a stack discipline - the engine ensures that the function
we are defining ought only to be called in the context of loading a
pipe, and these two vars are guaranteed to be set appropriately to the
pipe we're loading.

87 creates a do-sync, because *pipe* and *pipe-args* (now pipe# and
pipe-args#) are refs and we will be modifying them.

88 lets pipe-key# to the ID of the pipe, unless it is an "output"
component, in which case it gives it the special name of "pipe-
output". All pipes have a component named pipe-output, so the system
knows which one to pull from first.

Line 91 just adds a key/val to *pipe*. The key is the pipe-key# from
line 88. The value is a simple function that basically wraps the
action-fun# - the function we originally passed in - in some error-
handling code and passes it the *pipe-args* ref.

So - it looks complicated, but all that is happening is that we are
defining a function that we will call when _defining_ a pipe. The
function we define, when called, takes a ref called *pipe*, and adds a
key and a value, where the key is the ID of a component and the value
is the function we want to call when _running_ the component.

Sorry it's hard to explain more clearly in text. I can certainly go
over it next time we get together.

Thanks,
-Luke




On May 5, 7:15 am, Serge Wroclawski <emac...@gmail.com> wrote:
> On Mon, May 4, 2009 at 4:54 PM, Luke VanderHart
>
Reply all
Reply to author
Forward
0 new messages