Hi,
I want to suggest a "pipe" macro for dealing with collection streams,
as a one-line variation on the built in "->" macro.

Instead of writing a nested stream like this:
  ; "Take the first 3 elements of the odd numbers in the range 1 to
20"
  (take 3 (filter odd? (range 1 20)))

you can write this:
  ; "Take the range 1 to 20, filter the odd numbers, then take the
first 3"
  (pipe (range 1 20) (filter odd?) (take 3))

I think both forms have their place. The piped form seems useful when
you want to emphasise the collection, or when the chain gets very
long. It's also close to how I'd draw the chain on a whiteboard or
describe it in speech, and helps me to reason about the chain from
left to right in small "chunks".

The difference with "->" is this:
(-> x (f a b)) => (f x a b)
(pipe x (f a b)) => (f a b x)

The definition is the equivalent of the "->" macro (defined in
core.clj, line 984), swapping "~@(rest form) ~x" to "~x ~@(rest
form)":

(defmacro pipe
  "Threads the expr through the forms. Inserts x as the
  last item in the first form, making a list of it if it is not a
  list already. If there are more forms, inserts the first form as the
  last item in second form, etc."
  ([x form] (if (seq? form)
              `(~(first form) ~@(rest form) ~x)
              (list form x)))
  ([x form & more] `(pipe (pipe ~x ~form) ~@more)))

I've seen pipe written as a function with the use of reader macros, or
as macros built from scratch. I'm putting this one forward because the
tiny change in intention from "->" is reflected as a tiny delta in the
code.

Cheers,
Matt

This inspired me to write a general purpose version:

(defmacro let->
   "Provide a name that will be bound to the result of the first form.
   For each additional form, the variable will be
   used in the invocation, and then rebound to the result of the form."
   [varname start & forms]
   (let [fn-args `[~varname]
         wrapped (map (fn [form] `(fn ~fn-args ~form)) forms)]
     (reduce
           (fn [acc func] `(~func ~acc))
           start
           wrapped)))

Some examples:

Gorilla=> (let-> x (+ 1 2) (* 2 x) (+ x 1))
7
Gorilla=> (let-> x [1 2 3] (map inc x) (reduce + x) (+ x 3))
12
Gorilla=> (macroexpand-1 (quote (let-> x [1 2 3] (map inc x) (reduce +  
x) (+ x 3))))
((clojure.core/fn [x] (+ x 3)) ((clojure.core/fn [x] (reduce + x))  
((clojure.core/fn [x] (map inc x)) [1 2 3])))

I tried a more general version that allowed any binding form and  
multiple variables in the first position, but it never seemed as clean  
as the single variable version, so I reverted to the simpler case. I  
also considered using function literals and % notation, but decided  
that since you couldn't nest them and map/reduce/etc with function  
literals are very common, using an explicitly bound var would be the  
best option.

The doc string could probably be more clear. Suggestions and  
improvements welcome.

-Mark

On Feb 7, 2009, at 6:27 PM, MattH wrote:

- Mark Fredrickson
mark.m.fredrick...@gmail.com
http://www.markmfredrickson.com

Nice, I would definitely use this!

One comment/question: would it be more efficient to expand to a bunch
of nested "let" statements, rather than nested function calls?  I'm
not sure how Clojure handles "let" under the hood, or how Hotspot
inlining works here.  Here's my version:

(defmacro let->
   "Provide a name that will be bound to the result of the first form.
   For each additional form, the variable will be
   used in the invocation, and then rebound to the result of the
form."
   [varname start & forms]
   (if (empty? forms)
       start
    `(let [~varname ~start]
       (let-> ~varname ~@forms))))

In a few quick timing tests I see no real difference, so maybe it
doesn't matter.

-Jason

On Feb 9, 2:31 pm, Mark Fredrickson <mark.m.fredrick...@gmail.com>
wrote:

I like that implementation. The recursive call makes it much cleaner.  
A slight improvement (?) yet:

(defmacro let->
   "Provide a name that will be bound to the result of the first form.
   For each additional form, the variable will be
   used in the evaluation, and then rebound to the result of the
   form."
   ([varname start] start)
   ([varname start & forms]
    `(let [~varname ~start]
       (let-> ~varname ~@forms))))

As to fn vs. let. I originally started with nested lets, but was  
getting bogged down in more boiler plate (the recursive call solves  
the problem). Then I remembered the equivalency of these two forms:

Gorilla=> ((fn [x y] (+ x y)) 1 2)
3
Gorilla=> (let [x 1 y 2] (+ x y))
3

But just because the second can be expanded into the first, doesn't  
mean that is what is clojure is doing behind the scenes (and a quick  
investigation shows it is not).

I could argue it either way on the efficiency angle. On one hand it  
seems like a function call would be more expensive than pushing  
variables on the stack that clojure uses to keep track of bindings. On  
the other hand, I'm sure HotSpot can work wonders with tiny little  
inline-able functions in repeated code. My vote is for the cleaner  
implementation.

-Mark

On Feb 9, 2009, at 5:40 PM, Jason Wolfe wrote:

- Mark Fredrickson
mark.m.fredrick...@gmail.com
http://www.markmfredrickson.com

Yes, even better :)

Well, given that quick tests showed them to be within 5% of each-other  
on the examples you gave, maybe both end up the same after Hotspot  
does its magic.  In the absence of any other reasons, I'm with you in  
choosing the cleanest solution (i.e., the one you just posted).

-Jason

I like the pipe macro.  I get a bit cognitively overloaded when map/filter/reduce are nested, I think it is made worse because they have 2 or more arguments, so you have to do a lot of jumping around to follow them.  The left-to-right style is much easier to follow.

I'm not sure about let-> though.  I think it might be doing a bit too much.  I think it could be confusing that the 'x' in the first expression is a different 'x' than the 'x' in the second expression.

--
Dave

The pipe macro is definitely not a new idea btw. It's taken from a
thread posted on another lisp group.

Someone posted a silly inflammatory attack on lisp, contrasting unix:
"cat a b c | grep xyz | sort | uniq"
  to how they'd imagine it in lisp:
"(uniq (sort (grep xyz (cat a b c))))"

A poster called Edward retorted:
"
(pipe (cat a b c) (grep xyz) (sort) (uniq))
'Nuff said.
"

Here's the link, but *not* recommend reading :)
    http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/d0...

When watching the Abelson and Sussman (SICP) lectures online, when
Abelson was using map/filter/reduce etc, he would sometimes explain
the code from the inside out, pointing to the most inner part, then
working outwards and up.

I get the same cognitive overload on seeing lots of nesting. Expanding
out onto multiple lines helps, but you still hit a wall.

(transform-g
  (transform-f
  ...
          (transform-b)
              (transform-a)
                  (some-generator))))))))

  vs

(pipe
  (some-generator)
  (transform-a)
  (transform-b)
  ...
  (transform-f)
  (transform-g)
)

The nested form would tell me "it's time to generalise and write a new
function". But if using the pipe form I'd happily expand and edit the
chain of transforms. It's pretty easy to swap parts of the chain, or
disable transforms by commenting them out.

The let-> form is really interesting and a nice implementation -
pretty amazing to see solution to the general case so quickly. I have
the same reservation about using it though, I wouldn't want "x" to
mean something different in the same lexical scope. Maybe it's just a
case of finding a good name for "x"? Something like "the-evaluation-of-
the-previous-form", but shorter..

Maybe _ is appropriate?

=> (let-> _ (+ 1 2) (* 2 _) (+ _ 1))
7
=> (let-> _ [1 2 3] (map inc _) (reduce + _) (+ _ 3))
12

Or maybe ? ?

/mike.

Hi,

Am 10.02.2009 um 23:34 schrieb Michael Reid:

I would not use _. _ is often used as "don't care" variable
name. And this is certainly not what we want here.. ;)

Unfortunately % doesn't work, but ? does.

(-> (+ 1 2) (* 2 ?) (+ ? 1))

(If we make the arg explicit, we can also redefine
-> totally....)

Sincerely
Meikel

smime.p7s 5K Download

Don't forget the wide variety of unicode symbols you have at your  
disposal:

user=> (let-> ★ 2 (+ ★  3) (- 10 ★ ) (map #(* ★  %) [2 3 4]))
(10 15 20)

Perfectly valid. Hard to miss that. (It is a star glyph incase this  
doesn't come across).

Of course, this is matter of style and I would not want to enforce a  
specific character. Also, doing so could lead to unhygienic clashes of  
a nasty sort.

-M

+1 for something like (let->).

I don't imagine myself being confused by the reserved symbol being
bound to different values in each successive form, or, in any case,
would gladly trade that possibility for the advantage of being able to
place the special symbol anywhere in the forms.

Somewhat along the lines of what Miekel says, if the special symbol is
not user-defined, the macro could be named after the special symbol,
e.g. x->, or, (keeping with the the "threading the needle" metaphor in
->'s docs) ndl->, or with the pipe metaphor, P->. The latter has the
advantage that capitals aren't idiomatic for variable names in
Clojure.

In any case, I vote for approaching Konrad Hinsen about putting this
in clojure.contrib.macros when a naming convention is agreed on.

Perry

On 11.02.2009, at 18:18, Perry Trolard wrote:

When I started clojure.contrib.macros, I intended it as a repository  
for everybody's small macros that don't have any other obvious place.  
So I don't mind anyone on the clojure.contrib group adding whatever  
they seem fit.

BTW, I completely agree about the utility of the pipe macro, though I  
can't make up my mind about which syntax I would prefer. I remember  
that Paul Graham discusses such a macro in On Lisp, using "it" as the  
parameter, but I forgot what is macro is called.

I have a similar operation in my monad library, which is called m-
chain. It expects a list of monadic operations that are functions of  
one argument, the result being again a function of one argument  
representing the composite operation. Using this operation in the  
identity monad would yield something quite close to the pipe macro  
being discussed, but as a function. Example:

(def my-pipe
   (with-monad id
     (m-chain #(filter odd? %)  #(map inc %)))

(my-pipe (range 10))

The advantage of such an approach is that there is no new syntax  
rule: the parameter placeholder is the well-known %. Another  
advantage is that it can be written as a function. Of course, the  
drawback is that all those functions cause a run-time overhead. Plus  
a macro makes for shorter syntax, which is part of its purpose.

As you can see, I am fully undecided :-)

Konrad.

"Plus a macro makes for shorter syntax, which is part of its purpose."

Yeah the shorter syntax becomes even more apparent when you're piping/
threading through one-arg functions where it also saves on
parenthesis, as the "->" and "pipe" macros expand to a list if
necessary.

E.g. this code in a comment section in src/clj/clojure/zip.clj shows
someone clearly seeing the benefit of "->" for testing:
"
(def dz (vector-zip data))
[...]
(-> dz next next (edit str) next next next (replace '/) root)
[...]
(-> dz next next next next next next next next next remove (insert-
right 'e) root)
"

You could get similar benefits from pipe:
(pipe (whole-numbers) filter-even filter-prime filter-unlucky)

For the "let->" form, would it help to have the placeholder be part of
the macro's name?
E.g. "(->it (whole-numbers) (filter even? it))" or "(->% (whole-
numbers) (filter even? %))".
Looks more obvious, not sure of the implications though.

I think both the "pipe" and "let->" forms would be really useful
additions to "->". Names that make sense together would be plus.

"pipe->" and "let->" ?

Dare I suggest it <- might be an interesting 'name' for 'pipe' given
that -> calls in prefix and 'pipe' calls in postfix. But I'd rather
not see it contending with Datalog which <- is nice to use.  |  isn't
'official', but it works fine... > and < aren't 'officially' usable in
symbols either so why stop there? :) Overall I prefer Meikel's
suggestion of just having the one macro -> with ? appearing in lists.
For singular functions you wouldn't need to specify the ? so in most
use-cases it would be unchanged, and for multiple arity calls it would
be very explicit and clear what the desired operation was.

There seems to be a general consensus that some sort of explicit form  
of threading would be useful. Below are snippets of other messages,  
outlining the remaining points. I should point out that we are not  
limited to one macro/function to handle our needs. For my own part, I  
think as written let-> can meet most if not all needs (and I'll make a  
case for it below), but there is certainly the option of writing a  
family of threading functions/macros for more specialized purposes.

I believe Graham calls these "anaphoric" macros (http://www.dunsmor.com/lisp/onlisp/onlisp_18.html
). His examples tend to use a specified identifier ("it"), though I  
think the concept can be considered more broadly to be when a macro  
creates a binding that the user can use. (As compared to the macro  
creating a binding with a gensym that the user never sees). In  
Graham's case, the identifier is fixed ahead of time. In the case of  
let->, the user specifies the identifier.

I dislike this strategy for two related reasons: (1) It cannont be  
nested. Here is a pseudo-example:

(?-> 2 ... some computations ... (f (?-> ... I don't have access to  
the outer "?" anymore... )))

(2) Shadowing user bindings. If I bind ? to something in my code (e.g.  
(let [? (pred? foo)] (....))) the ?-> form shadows my definition,  
preventing access.

By asking the user for an explicitly named identifier, both of these  
issues can be avoided. This is not to say a ?-> form shouldn't be  
allowed in the family of thread functions, just that let-> as written  
as a distinct purpose.

On to functions and the % symbol. To begin with, I'm not sure there is  
any additional runtime overhead of the functional version. Just as  
many functions are created and called in them macro version. There are  
differences. The first is stylistic: The macro saves key strokes for  
longer pipes in the form of fewer # and/or (fn [x] ....) definitions.  
Not a big point. The second issue is that the functional version can't  
nest function literals. In the example, we see #(map inc %). That is  
all well and good if you are using predefined single arity functions.  
I tend to use function literals in (map) all the time. But the  
following is invalid: #(map #(+ 1 %) %). Using explicit variables,  
e.g. x, would give us (map #(+ 1 %) x), which is valid. I think this  
is a very important benefit of the macro form.

I'm not entirely clear on this proposal. It does prompt me to consider  
a more complicated macro that inspects its arguments and provides  
several behaviors under one name. If the first arg is a symbol, it  
works like let->. If the first argument is a list, it works like ->.  
If the first argument is the keyword :last it behaves like pipe. This  
would be backwards compatible with the current -> implementation, but  
also allow for the additional functionality described in this thread.  
I think a good implementation would be to have the three behaviors in  
their own macros, and have the super macro dispatch to the appropriate  
version (mullti-macros anyone?).

Thoughts?

- Mark Fredrickson
mark.m.fredrick...@gmail.com
http://www.markmfredrickson.com

Hi,

Am 12.02.2009 um 16:15 schrieb Mark Fredrickson:

These are valid points. And I think you are right in that (in case there
is an easy solution) there shouldn't be arbitrary restrictions.

I wouldn't make things to complicated. I think that eg. -> can be well
handled with this same macro by simple specifying a local at the
right place.

However I would not name it let-> since there is no binding vector
following. I would suggest pipe-as. This also explains, that the given
local is bound to different things during the different stages of the
pipe.

What I definitively want is the non-seq => (non-seq local) translation.
That's really nice.

For the function vs. let discussion: I use anonymous functions a lot in
macros to package things up in a closure and then expand the macro to
some code calling a driver function passing said anonymous function at
runtime. However, in this case it is not really needed. Let is
sufficient. And since each function generates a new classfile, while
this does not happen for let, I would stick with the latter.

user=> (defmacro pipe-as
          ([_ expr] expr)
          ([local expr form & more]
           `(let [~local ~expr]
              (pipe-as ~local
                       ~(if (seq? form) form (list form local))
                       ~@more))))
#'user/pipe-as
user=> (pipe-as x (+ 1 2) (* 3 x) inc)
10

user=> (defmulti foo* (fn [x a b] x))
#'user/foo*
user=> (defmethod foo* 'add [_ a b] `(+ ~a ~b))
#<MultiFn clojure.lang.MultiFn@e6f711>
user=> (defmethod foo* 'sub [_ a b] `(- ~a ~b))
#<MultiFn clojure.lang.MultiFn@e6f711>
user=> (defmacro foo [x a b] (foo* x a b))
#'user/foo
user=> (macroexpand-1 '(foo sub 1 2))
(clojure.core/- 1 2)
user=> (macroexpand-1 '(foo add 1 2))
(clojure.core/+ 1 2)

;)

Sincerely
Meikel

smime.p7s 5K Download

On Feb 12, 12:15 pm, Meikel Brandmeyer <m...@kotka.de> wrote:

Agreed, Mark. I thought these limitations were acceptable given that
compactness is the goal of the macro -- i.e. it wouldn't make much
sense to nest with it -- but you're likely right that it'd be
shortsighted.

+1 to Miekel's pipe-as: I agree that "let" forms should have a binding
vector.

Agreed.

Mark, I don't think this is actually backwards compatible. If the
first arg is a symbol, you don't know if it's a symbol to be bound or
if it's the initial value, e.g.

user=> (-> 0 inc dec)
0
user=> (def z 0)
#'user/z
user=> (-> z inc dec)
0

At the moment, my vote is for two macros:

1 pipe: like -> but accumulated value is place at end of each form

2 pipe-as: in which user specifies binding symbol for accumulated
value, plus the non-seq to (non-seq local) translation

Perry

Let me amend:

Instead of pipe-as, I think the symbol-binding version of the macro
should be derived from the more common macro, which is -> & not pipe.
So I'd vote for pipe & a let-> that takes a bindings vector as its
first argument (sample implementation below).

Perry

(defmacro let->
  "Like ->, but takes a binding form as first argument, the bound
symbol(s) from
  which will carry the resulting value from form to subsequent form.
E.g.
    (let-> [a 1] (str "Your number is:" a) (.replace a "Your"
"My") .toUpperCase) =>
   \"MY NUMBER IS: 1\""
  ([[sym init] form]
    (if (seq? form)
      `(let [~sym ~init] ~form)
      (list 'let [sym init] (list form sym))))
  ([[sym init] form & more]
    `(let-> [~sym (let-> [~sym ~init] ~form)] ~@more)))

+1 for pipe and let-> as above, with the non-seq => (non-seq local)
translation.

+1 as well for pipe and let->

Sorry to keep revising things; here's an improved version of let->,
which better supports (as near as I can tell -- please try to break)
arbitrary let-style destructuring. (The previous version favored
single-symbol bindings, but I can imagine use cases where [a b :as
all] kinds of destructuring are useful.)

Perry

(defmacro let->
  "Like ->, but takes a binding form as first argument, the bound
  symbol(s) from which will carry the resulting value from each form."
  ([[bind init] form]
    (if (seq? form)
      `(let [~bind ~init] ~form)
      (list form init)))
  ([[bind init] form & more]
    `(let-> [~bind (let-> [~bind ~init] ~form)] ~@more)))

That's very neat Perry!

user=> (let-> [? 5] inc)
6
user=> (let-> [? 5] inc (+ 2 ?))
8
user=> (let-> [? 5] inc (+ 2 ?) (+ ? 3))
11
user=> (let-> [? 5] inc (+ 2 ?) (+ ? 3) (+ 4))
4

What should happen when/if the seq arg doesn't contain the symbol? I
believe how you currently handle it is correct and in the spirit of
let-> (alternatively it could be reported as an error) however it may
raise yet another possibility for pipe:
(pipe 5 inc (+ 2) (+ ? 3) (+ 4 ? 2))
ie: if the argument is a seq and doesn't contain ? then it is assumed
to be a post argument, but if it does contain ? then can be explicitly
a pre or mid argument. This is however subject to the previously
discussed issues of nesting and rebinding (are they realistic?) having
a mandatory binding for pipe would be a burden so better to forget
about it. Certainly pipe and let-> as proposed allow for all cases
very nicely.

Is it worth considering how (doto) fits into the picture? My initial
observation is that (doto) is orthogonal in the sense that its primary
use is for java object manipulation, and there is never a use case to
have arguments in different places for that. So I suspect doto is
irrelevant.

Hi Timothy,

On Feb 12, 8:08 pm, Timothy Pratley <timothyprat...@gmail.com> wrote:

The problem that I see with scanning for the special symbol &
inserting it when not found is that it really requires a single symbol
as the binding. But I think it's good for let-> to behave just like
let does -- no surprises -- which means arbitrary destructuring. So,
for example, if my binding is like

  (let-> [[a :as all] coll]
     (if (pred? a) (map func1 all) (map func2 all))
     some-final-func)

what should happen if the last form was (map some-final-func) instead
of some-final-func? Which symbol should be inserted into the form?

One answer is that instead of a symbol being inserted, the value would
be (this is what happens when the form isn't a list, .e.g. (let-> [a
"string"] .toUpperCase) => (.toUpperCase "string")). But that brings
the question of *where* to put it -- second, or last? We'd have to
decide to follow -> or pipe, & I don't know that one makes more sense
than the other.

I think it's most straightforward to require that one manually places
the symbol for all cases other than the non-list.

I agree -- doto assumes methods that mutate an object, & I think those
are always going to be of the form (.method object [args]).

Best,
Perry