core.async - restriction on macros inside go block?

[Tom Locke]

Hi All,

I had a terrible time trying to write a convenience macro for reading from a core.async channel.

The idea was to make it easier to do the common "dispatch on first part of a message" pattern, as in:

(let [[msg & args] (<! my-channel)]
(case msg
:msg1 (let [[a b] args] ...)
:msg2 (let [[x] args] ...)))

After a long struggle I came to the conclusion that there is some problem with using macros inside a go block, or at least using <! inside a macro. Is that right? Is this documented anywhere?

Thanks

Tom

[Daniel Kersten]

You must use <! and >! from within go directly - you cannot use them inside a function or macro.

This is because the go macro walks the abstract syntax tree to turn it into a state machine and as it does this, it turns the calls to <! and >! into something else (in fact, if you look at the source, you will see that <! and >! themselves actually don't do anything! [1], because the go macro replaces it with the real code).

The macro doesn't know how to walk through function or macro calls, so cannot find <! and >! contained within and therefore cannot turn them into the proper code.

The only way you could do this is do something similar to the go-loop macro: have your macro return a go macro AFTER walking the code passed in replacing your special dispatch code with normal calls to <!.

Or, what I do: encapsulate my go block in a function which takes as arguments functions to dispatch to (but the function itself handles the go block and dispatch, so I don't need to write this each time).

[1] The ClojureScript source for <! is:

(defn <!


  "takes a val from port. Must be called inside a (go ...) block. Will
  return nil if closed. Will park if nothing is available.

  Returns true unless port is already closed"


  [port]
  (assert nil "<! used not in (go ...) block"))

Tom

--
Note that posts from new members are moderated - please be patient with your first post.
---
You received this message because you are subscribed to the Google Groups "ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojurescrip...@googlegroups.com.
To post to this group, send email to clojur...@googlegroups.com.
Visit this group at http://groups.google.com/group/clojurescript.

[Tom Locke]

Thanks for your reply. That's pretty much what I'd come to understand. I'm a bit puzzled why this restriction exists though. Of course function calls can't be resolved at macro expansion time, so it makes sense that >! & <! inside functions can't be found. But I don't understand why the go macro cannot first expand all nested macros and then apply its transformations to the result.

Indeed, clojure.walk contains macroexpand-all specifically for this kind of situation, or that's what I had understood.

[Daniel Kersten]

There's really two questions in there as I see it.

1. Why doesn't the go macro expand all nested macros.

2. Why can't the go macro find the <! and >! anyway

I don't know the answer to either, unfortunately.

Because macro expansion happens in reverse order to normal evaluation, nested macros aren't normally expanded. Why it doesn't manually expand nested macros, I don't know, but I guess perhaps its to avoid compile-time overhead to using core.async if it has to macroexpand-all every form inside a go block?

I do remember seeing someplace that Rich or others may have purposely left it as it is to force channel operations to always happen at the top level. This makes sense, if true, because nesting the <! and >! operations means that you lose a lot of purity and the code (not to mention the core.async code) would easily get complicated. But as you say, while this applies to functions, macros should really live outside that restriction.

As for #2, I haven't a clue. I mean, surely if go does not expand the macros it would walk the tree and find the <! and >! anyway. I didn't think about this yesterday, but now I wonder why it doesn't.

On 22 July 2014 15:05, Tom Locke <t...@tomlocke.com> wrote:

Thanks for your reply. That's pretty much what I'd come to understand. I'm a bit puzzled why this restriction exists though. Of course function calls can't be resolved at macro expansion time, so it makes sense that >! & <! inside functions can't be found. But I don't understand why the go macro cannot first expand all nested macros and then apply its transformations to the result.

Indeed, clojure.walk contains macroexpand-all specifically for this kind of situation, or that's what I had understood.

[Kevin Marolt]

Not sure if I understood the initial question correctly, but isn't the following sort of what you wanted?

;; ======

(require '[clojure.core.async :as async])

(defmacro dispatch-on [ch & cases]
(let [argssym (gensym "args__")
keysym (gensym "key__")
ressym (gensym "res__")
default-case (when (odd? (count cases)) (last cases))
default-case (when default-case
(let [[[res] & body] default-case]
`(let [~res ~ressym] ~@body)))
cases (if default-case (butlast cases) cases)
case-body (apply concat (for [[key [args & body]] (partition 2 cases)]
[key `(let [~args ~argssym] ~@body)]))
case-body (if default-case
(conj (vec case-body) default-case)
case-body)]
`(let [ch# ~ch
~ressym (async/<! ch#)]
(if (and (vector? ~ressym) (seq ~ressym))
(let [~keysym (first ~ressym)
~argssym (next ~ressym)]
(case ~keysym ~@case-body))
~default-case))))

(def ch (async/chan))

(async/go
(loop []
(dispatch-on ch
:add ([a b] (println a "plus" b "is" (+ a b)) (recur))
:sqrt ([x] (println "The square root of" x "is" (Math/sqrt x)) (recur))
;default
([x] (when-not (nil? x)
(println "unknown operation:" x)
(recur))))))

(async/put! ch [:add 40 2])
(async/put! ch [:sqrt 1764])
(async/put! ch [:foo :bar])
(async/put! ch :foobar)
(async/close! ch)

;; =====

This prints:

40 plus 2 is 42
The square root of 1764 is 42.0
unknown operation: [:foo :bar]
unknown operation: :foobar

While it's true that the go-macro can't peek beyond a function's boundaries, it actually *does* perform macroexpansion before applying its transformations.

Cheers,
Kevin

[Daniel Kersten]

Ah, I should have just tried it before answering :)

After I thought about it, it certainly does make sense that it would either expand the macros or at least see the code within (though I guess there are issues there (#2 in my last email) is it shouldn't assume that what is seen in a macro is normal code). I don't know what Tom's initial problem is though if macros do get expanded.

[Tom Locke]

On Tuesday, 22 July 2014 16:42:00 UTC+2, Kevin Marolt wrote:
> Not sure if I understood the initial question correctly, but isn't the following sort of what you wanted?

Yes that's exactly what I was after, although that is Clojure, right? I've tried it in ClojureScript (not much to change - just requiring the right namespaces / macros) and it doesn't work. I get:

No implementation of method: :emit-instruction of protocol: #'cljs.core.async.impl.ioc-macros/IEmittableInstruction found for class: cljs.core.async.impl.ioc_macros.Jmp

I've tried playing around moving the <! and the (recur) outside of the on-dispatch macro, and, while I can avoid the above error and get it to compile, the behaviour of the test code makes no sense at all. It seems like mixing the go macro with your own macros is very fragile in cljs.

Tom

[Kevin Marolt]

I tried it in ClojureScript and I did get the same error as your did. I believe this is because the go-macro doesn't handle the case-statement correctly (or, rather, not at all). Switching from (case key ...) to (condp = key ...) however appears to do the trick (along with all the other awkward namespacing stuff):

(defmacro dispatch-on [ch & cases]
(let [argssym (gensym "args__")
keysym (gensym "key__")
ressym (gensym "res__")
default-case (when (odd? (count cases)) (last cases))
default-case (when default-case
(let [[[res] & body] default-case]

`(cljs.core/let [~res ~ressym] ~@body)))

cases (if default-case (butlast cases) cases)
case-body (apply concat (for [[key [args & body]] (partition 2 cases)]

[key `(cljs.core/let [~args ~argssym] ~@body)]))

case-body (if default-case
(conj (vec case-body) default-case)
case-body)]

`(cljs.core/let [ch# ~ch
~ressym (cljs.core.async/<! ch#)]
(if (cljs.core/and (cljs.core/vector? ~ressym) (cljs.core/seq ~ressym)) ;; <- still problematic
(cljs.core/let [~keysym (cljs.core/first ~ressym)
~argssym (cljs.core/next ~ressym)]
(cljs.core/condp = ~keysym ~@case-body))
~default-case))))

However, this still gives an error if you put :foobar onto the channel. This, too, appears to be a core.async bug: in the line highlighted above, *both* arguments to "and" get evaluated, regardless of whether the first argument is false or not.

[Kevin Marolt]

This should circumvent the issue with "and":

(defmacro dispatch-on [ch & cases]
(let [argssym (gensym "args__")
keysym (gensym "key__")
ressym (gensym "res__")
default-case (when (odd? (count cases)) (last cases))
default-case (when default-case
(let [[[res] & body] default-case]
`(cljs.core/let [~res ~ressym] ~@body)))
cases (if default-case (butlast cases) cases)
case-body (apply concat (for [[key [args & body]] (partition 2 cases)]
[key `(cljs.core/let [~args ~argssym] ~@body)]))
case-body (if default-case
(conj (vec case-body) default-case)
case-body)]
`(cljs.core/let [ch# ~ch
~ressym (cljs.core.async/<! ch#)

isvec# (cljs.core/vector? ~ressym)]
(if (cljs.core/and isvec# (cljs.core/seq ~ressym))

[Kyle Cordes]

On Monday, July 21, 2014 at 12:28 PM, Daniel Kersten wrote:
> You must use <! and >! from within go directly - you cannot use them inside a function or macro.
>
> This is because the go macro walks the abstract syntax tree to turn it into a state machine and as it does this, it turns the calls to <! and >! into something else (in fact, if you look at the source, you will see that <! and >! themselves

To me, this limitation ends up as a an important hole in the otherwise compelling story of Clojure extensibility via macros in general and of core.async in particular. It feels like the (important) go construct in core.async is plugged into Clojure as a macro because that is the only plug point available that is somewhere near its needs.

But to be more complete, core.async go would need to fit in a different kind of plug point. It would act as something like a macro that plugs in after parsing of an entire file (or an entire program?), after other macros, before the next step of compilation. Now this sounds perhaps close to a “reader macro” which can make arbitrary changes to the language even up the syntax level; but from my (very shallow relative to the gurus) understanding, I think that is not really the case. I think such a plug point would be still in keeping with the notion that Clojure code always looks like Clojure code and generally does approximately what it looks like modulo some limited modification via macros.

Of course it is easy to sit here in comfort and describe a deep, complex, and important change someone else could make. :-)

--
Kyle Cordes

http://kylecordes.com

[Tom Locke]

> the go-macro doesn't handle the case-statement correctly

I've had a look on the core.async jira and can only see one open issue that looks like it could possibly be related, with the .. macro

http://dev.clojure.org/jira/browse/ASYNC-49

I'm going to work on a minimal test case and file a new issue.

@Kyle - it's not a deep problem with Clojure extensibility but just a bug in the ClojureScript implementation.

Tom

[Tom Locke]

What I forgot to mention was that (case) seems to work fine directly inside a go block, but not within a macro, so it seems to be a bug related to macros inside go blocks. That's why I think the .. issue (49) could be related.

[Tom Locke]

OK I've filed this here:

http://dev.clojure.org/jira/browse/ASYNC-79

In trying to create a minimal test case I tried a trivial macro

(defmacro my-case [expr & cases] `(case ~expr ~@cases))

inside a go block, and that worked fine, so it's more subtle than just "case in a macro is broken".

I didn't try to reproduce the (and) bug spotted by Kevin M.

The thing that's still confusing to me, is why the go macro doesn't just macroexpand the body before it does it's thing.

[Kyle Cordes]

On Wednesday, July 23, 2014 at 2:06 AM, Tom Locke wrote:
> @Kyle - it's not a deep problem with Clojure extensibility but just a bug in the ClojureScript implementation.
>

I agree it is not a problem per se; but the feature I would really like, which I believe would require the approach I described, would be to allow the full set of Clojure constructs to be composed arbitrarily with “go”. For example a <! could live in a function, called by another function, called via a protocol, via a macro, called itself via a multi method, in another namespace, in another file, which happens to be called via a go somewhere.

(And I’d like a pony.. :-) )

[Daniel Kersten]

Honestly, I'm against having core.async as anything other than a normal macro. The less built-in magic the better, plus its nice to know that almost everything I use, I could in theory build myself without hacking the language itself, should I ever want to. The more features that get special treatment by the compiler or runtime, the less this is true.

Also, while I'm all for allowing macros to transform code in go blocks that contains <! and >! (and it seems in Clojure you can do this and this is merely a bug in cljs), I personally think that disallowing <! and >! in functions (outside of the technical limitations that we currently have) is not necessarily a terrible thing.

I mean, sure, it would be nice to lift restrictions so that more powerful abstractions can be built on top and I would certainly be all for doing so, should somebody figure out a way to avoid the current technical limitations, but if this were to happen, I think everyone should still be very strongly discouraged from using <! and >! anywhere other than in the go block. It keeps the code simple by forcing the bulk of it to be nice pure functional code (at least in respect to channel operations) - the Clean, essentially - and this greatly helps understanding and testing of code IMHO.

I guess, I would love for the ability only to build better abstractions, not for day to day use. But perhaps macros serve that purpose well enough already? (once the cljs bug is fixed, at least)

[Daniel Kersten]

*I meant to write "clean architecture" :-)

[Tom Locke]

I've discovered an easy workaround for this problem. During macro-expansion core names like case become fully qualified, i.e. cljs.core/case, and it seems that the go macro then fails to recognise the case as such. Replacing case with ~'case in the definition of let-case fixes the problem.

I haven't tried it but I wouldn't be surprised if using ~'and would also solve the problem with (and ...) noted above.

It might even be a solution for http://dev.clojure.org/jira/browse/ASYNC-49

[Daniel Kersten]

Nice detective work! That certainly clears things up a bit.