Having trouble doing what I want using macros, is there a better way?

[Travis Daudelin]

Hi all!

I'm current working on a project where I am ingesting events off a stream and processing them. There are many many steps involved in the processing component, so I am choosing to write the steps as a series of transducers (because, hey, they're super cool!). Here's the problem though, after writing the first 2 processing steps I'm noticing that all of them are going to look very similar:

(defn a-step-transducer
  []
  (fn [reducing-fn]
     (fn
       ([] (reducing-fn))
       ([result] (reducing-fn result))
       ([[guid-state processed-events :as result] event]
        ;; step-specific logic
))))

Given how many steps I am planning to write, this is a ton of boilerplate! So, my first thought was to use a macro to abstract away all this boilerplate. Now, I have to admit that Clojure is my first Lisp, so I'm really not sure I fully understand when or why to use macros to do things. My current understanding is that macros are a kind of "template" for code, so something like this where I don't want to write the same function structure over and over seems like a decent use case for macros (feel free to correct me if I'm totally off on this). Here is my first attempt:

(defmacro deftransducer
[body]
`(fn [reducing-fn]
   (fn
     ([] (reducing-fn))
     ([result] (reducing-fn result))
     ([[guid-state processed-events :as result] event]
      ~@body))))

The idea here being that in body I can reference the variables defined by the macro like reducing-fn, result, event, etc. Of course, I quickly found out that this doesn't work:

storm-etl.entry-processing> (deftransducer "something")
CompilerException java.lang.RuntimeException: Can't use qualified name as parameter: storm-etl.entry-processing/reducing-function

Some quick googling tells me that the solution to this is to use gensyms for these variable names, but that would defeat the whole purpose of this because I want to be able to reference those variables from within the code that I pass to my macro. Is this an appropriate use case for macros or am I way off base? Is there an alternative approach that would be recommended?

[Timothy Baldridge]

I think you can do what you want with existing transducers. Won't map/filter/keep/etc do the trick?

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

“One of the main causes of the fall of the Roman Empire was that–lacking zero–they had no way to indicate successful termination of their C programs.”
(Robert Firth)

[Raoul Duke]

My $0.02 is only resort to macros when all else has failed. Can just higher order functions and composition and injection get you closer to what you want?

[Nate Young]

You can "capture" symbols from the surrounding context, making them available to the body of your macros, the "tilde tick trick" is what you're looking for there:

----------------------------------------
(defmacro deftransducer
  [body]
  `(fn [reducing-fn#]
     (fn
       ([] (reducing-fn#))
       ([result#] (reducing-fn# result#))
       ([[~'guid-state ~'processed-events :as ~'result] ~'event]
        ~@body))))
----------------------------------------

now your macro's body has some implicitly defined variables available to it called `guid-state` `processed-events` `result` and `event`

If you'd like to avoid the tilde trick and name your own variables, a common pattern in Clojure is to pass in the binding form yourself as part of the macro like so (assuming you don't care much about the first two arities):

----------------------------------------
(defmacro deftransducer
  [params body]
  `(fn [reducing-fn#]
     (fn
       ([] (reducing-fn#))
       ([result#] (reducing-fn# result#))
       (~params ~@body))))
----------------------------------------

Now you can use this without being restricted to the same names every time:

-------------------------------------------------
(deftransducer [[my-state already-processed :as result] next-event]
  ...body goes here...)
----------------------------------------

Macros are great and lots of fun to write! But I think I'd echo what's already been thrown out on this thread and suggest that you try to recast your logic in terms of existing transducers or write a higher-order function: something that takes a processing function and returns a transducer.

Travis Daudelin

June 10, 2016 at 1:06 PM via Postbox

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Sent from Postbox

[Bobby Eickhoff]

Having spent quite a bit of time recently dissecting transducers, I'd tend to agree with Tim: core transducers will probably give you most of what you want.  I'd also agree that writing macros should be your last resort.  But maybe the core function completing is very close to what you're looking for...

clojure.core/completing

([f] [f cf])

  Takes a reducing function f of 2 args and returns a fn suitable for

  transduce by adding an arity-1 signature that calls cf (default -

  identity) on the result argument.

[Travis Daudelin]

I think you can do what you want with existing transducers. Won't map/filter/keep/etc do the trick?

I can't say for sure that it's not possible, but I certainly lack the imagination :). The logic I need to write is quite complicated and I'm finding it's easier to write my own transducer to have fine-grained control.

My $0.02 is only resort to macros when all else has failed. Can just higher order functions and composition and injection get you closer to what you want?

This seems to be very common feedback on this group, and I'll definitely take it to heart. Mostly I just wanted to check my understanding of macros and verify that what I want to do is / is not possible using them before moving on to higher order functions.

You can "capture" symbols from the surrounding context, making them available to the body of your macros, the "tilde tick trick" is what you're looking for there

Ah perfect! This is exactly what I was hoping would be possible. Your examples are right on point, thanks for educating me :)

[Travis Daudelin]

On Friday, June 10, 2016 at 11:43:04 AM UTC-7, Bobby Eickhoff wrote:

But maybe the core function completing is very close to what you're looking for...

Hmm, looking through its source I'd say it's exactly what I'm looking for. Thank you! 

[Travis Daudelin]

Actually, I spoke too soon. It looks like completing takes in a reducing function and wraps it so that it meets the arity expectations of a transducer. While this is still super useful to my needs (thanks again!) I wanted to clarify for posterity that completing does not solve the issue in my initial post.

[Francis Avila]

A higher-order function can do what this macro does: https://gist.github.com/favila/ecdd031e22426b93a78f

[Travis Daudelin]

On Friday, June 10, 2016 at 3:03:38 PM UTC-7, Francis Avila wrote:

A higher-order function can do what this macro does: https://gist.github.com/favila/ecdd031e22426b93a78f

Oh nice! It looks like I came up with an almost identical solution:

(defn transducing
  [f]

  (fn [reducing-fn]
    (fn
      ([] (reducing-fn))
      ([result] (reducing-fn result))

      ([result input] (f result input reducing-fn)))))

It feels like "writing a custom transducer" should be a common use case, is there anything like these functions we've written that exists in the core library? I didn't see anything like them there or in the reducers library when I first started on my project.

[Timothy Baldridge]

Touching the accumulator (the result in your case) from within a transducing function is a bit of an anti-pattern. The whole point of transducers is that you can swap out the accumulator:

(transduce (map inc) conj [] (range 10))

(transduce (map inc) async/>! some-channel (range 10))

(transduce (map inc) + 0 (range 10))

In this example, we loose that ability to swap out the result type if `(map inc)` assumes that the accumulator will always be a vector, or a integer. This is why I said map/filter/keep should handle most of your cases. 

If the function you are writing doesn't require the accumulator to function properly, then there is no reason why `(map my-fn)` won't work. 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Francis Avila]

These functions (as-transducer, transducing) are still completely agnostic about the type of the accumulator and result as long as the function you give them only touches result in the following ways:

1. Return result unchanged

2. Return (xf (xf result X) Y), I.e. Whatever you get from applying xf to result with a value to add, one or many times. (In case result is mutable, eg a channel, or the transformation is stateful, never call xf unless you plan to use the return value.)

3. (reduced result)

4. Both 3 and 4

The advantage of writing a full transducer (or as-transducer/transducing) is that the step arity is truly a reduction-step function and so can efficiently not change the result, add multiple items, or short-circuit reduction (or even a different thing each time).

Normally data transformations don't do more than one of these at a time, so map or mapcat or filter is fine.

But when the transduction is modeling a process (I.e read input, possibly keeping old state, and unpredictably emit 0, 1, many results based on the input) it is handy to be able to do exactly what you mean instead of using mapcat/map/filter/keep etc with intermediate collection or sentinel value contortions. It's also nice not to have an intermediate collection (mapcat) if you need to reduce object allocations.

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/-XIekEB6zu0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.