Confusing interplay between macros and metadata

114 views
Skip to first unread message

Alan Malloy

unread,
Oct 24, 2011, 2:54:35 PM10/24/11
to Clojure
I'm struggling with a basic feature of how macros behave. I understand
how the
problem arises, and I can cobble together my own fix in the specific
places
where it's causing me trouble, but it seems like a prettier, more
general
solution would be desirable. Below is a brief transcript demonstrating
the
problem.

user> (defmacro call [f arg] `(~f ~arg))
#'user/call
user> (let [f inc] (.intValue (f 10)))
Reflection warning, NO_SOURCE_FILE:1 - reference to field intValue
can't be resolved.
11
user> (let [f inc] (.intValue ^Integer (f 10)))
11
user> (let [f inc] (.intValue ^Integer (call f 10)))
Reflection warning, NO_SOURCE_FILE:1 - reference to field intValue
can't be resolved.
11

I want to typehint the return value of f, so I put metadata on the
form
representing a call to it. But if a macro gets involved, there's an
"intervening" form that ignores its metadata and returns a new list of
'(f 10)
with no metadata. Thus the compiler has no idea I ever wanted to give
it a hint
about the type.

There are two solutions that are simple enough for me to apply:

(1) At the call site I can bind the result of (call f 10) to a local
named i and
then put the typehinting metadata on that

(2) I can edit the call macro to return a form with the right
metadata:
(defmacro call [f arg] (with-meta `(~f ~arg) (meta &form)))

Both of these work, but they seem awful. If the language specifies
you're
supposed to be able to typehint expressions as well as named bindings,
it's both
unintuitive and quite inconvenient that most macros do not "respect"
this
behavior by default. And many macros I don't have enough control over
to make
this change. For example, the whole issue arose when I was trying to
hint the
result of a (for ...) as a java.util.List. It ignores my metadata and
returns a
new form; and I certainly can't go edit its source, so instead I have
to bind
the result in a let, for no reason other than to typehint it.

It seems to me that it would be nice to have macros automatically
include, on
their result forms, the metadata from their input &form. Of course,
macros may
wish to add metadata as well, so the two maps should probably be
merged. However, there are certainly some problems with this approach:
for
example if a macro wants to return something that can't suppport
metadata (like
an Integer), the compiler needs to be careful not to try to include
it. So I'm
hoping the community can comment on whether this feature would be
useful, or
whether there are fundamental problems with it that I haven't
foreseen. Is there
a reason this can't make it into a future version of Clojure?

Kevin Downey

unread,
Oct 24, 2011, 3:37:59 PM10/24/11
to clo...@googlegroups.com
it's not a macro issue, it's a syntax quote issue

> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com
> Note that posts from new members are moderated - please be patient with your first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
>

--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

Alan Malloy

unread,
Oct 24, 2011, 3:43:05 PM10/24/11
to Clojure
I'm not sure I buy that. If I write my macro as (defmacro call [f arg]
(list f arg)), which I did consider doing for this post, the same
thing happens. Perhaps you could explain what you mean?

Kevin Downey

unread,
Oct 24, 2011, 3:50:46 PM10/24/11
to clo...@googlegroups.com
syntax quote effectively does that it, it rewrites your forms as a
series of calls to various sequence functions like concat and in the
shuffling it loses metadata.

Alan Malloy

unread,
Oct 24, 2011, 3:56:44 PM10/24/11
to Clojure
That would be relevant if I were talking about losing metadata on
arguments to the macro, say like (call .length ^String (identity
"test")). But I'm talking about metadata on &form - syntax-quote never
sees that at all, so there's nothing for it to lose.

Marshall T. Vandegrift

unread,
Oct 25, 2011, 5:27:41 AM10/25/11
to clo...@googlegroups.com
Alan Malloy <al...@malloys.org> writes:

> It seems to me that it would be nice to have macros automatically
> include, on their result forms, the metadata from their input
> &form. Of course, macros may wish to add metadata as well, so the two
> maps should probably be merged. However, there are certainly some
> problems with this approach: for example if a macro wants to return
> something that can't suppport metadata (like an Integer), the compiler
> needs to be careful not to try to include it. So I'm hoping the
> community can comment on whether this feature would be useful, or
> whether there are fundamental problems with it that I haven't
> foreseen. Is there a reason this can't make it into a future version
> of Clojure?

I think this is an excellent idea. Overall I believe this reduces the
number of situations in which one needs to be actively aware that a
particular expression will be subject to macro-expansion. The
alternative-world where the 99% of well-behaved macros returning IMetas
manually forward metadata from &form seems like a world with way too
much boilerplate to me.

Forwarding metadata does preclude macros which use metadata applied to
&form as parameters and construct new metadata which may not include the
literal values specified in the user-supplied metadata. I'm not sure
this is a good idea anyway, but this sort of case -- and any others
where metadata should not be forwarded/merged -- could easily be
supported by providing a variation of defmacro with the current
behavior. Or perhaps by configuring the metadata-forwarding behavior
via metadata on the macro var -- something like:

(defmacro foo {:forward-meta false} ...) ;; or,
(defmacro ^:replace-meta foo ...)

If a macro expands to something which doesn't implement IMeta, then I
believe the compiler needs to error out if metadata is applied to it,
just as it does applying metadata to non-IMeta literals. To do
otherwise would be inconsistent, and result in the same silent data-loss
as macros are yielding today.

This proposal doesn't touch forwarding metadata on forms which become
macro arguments, obviously. There's still room for inconsistency there,
but I think that's clearly in the court of individual macro authors to
implement the correct behavior.

Are there other contexts where metadata should potentially be forwarded?
I don't know how often this comes up, but:

(= (meta ^:foo (quote foo)) (meta (quote ^:foo foo)))
;; => false

-Marshall

Alan Malloy

unread,
Oct 26, 2011, 4:21:25 PM10/26/11
to Clojure
On Oct 25, 2:27 am, "Marshall T. Vandegrift" <llas...@gmail.com>
wrote:
I filed a ticket and wrote a patch: it's at http://dev.clojure.org/jira/browse/CLJ-865
if you're interested. As for your last snippet, I suppose it might
conceivably make sense to do this forwarding but I think it's
preferable to just understand what reader metadata and quoting mean,
since the difference between these is a fairly necessary consequence
of those things.

Tassilo Horn

unread,
Nov 8, 2011, 3:43:28 PM11/8/11
to clo...@googlegroups.com
Alan Malloy <al...@malloys.org> writes:

Hi Alan,

> I want to typehint the return value of f, so I put metadata on the
> form representing a call to it. But if a macro gets involved, there's
> an "intervening" form that ignores its metadata and returns a new list
> of '(f 10) with no metadata. Thus the compiler has no idea I ever
> wanted to give it a hint about the type.

I'm facing the same issue. I have this macro for java interop:

--8<---------------cut here---------------start------------->8---
(defmacro with-traversal-context
[[g tc] & body]
`(let [old-tc# (.getTraversalContext ^Graph ~g)]
(try
(.setTraversalContext ^Graph ~g ^TraversalContext ~tc)
~@body
(finally (.setTraversalContext ^Graph ~g ^TraversalContext old-tc#)))))
--8<---------------cut here---------------end--------------->8---

But the type hints are gone in the macro expansion, thus I have 3
reflection warnings per macro application, and real, performance
critical reflection warnings get lost in the shuffle.

Is there a way to suppress reflection warnings only in a given scope,
i.e., something like a `do' in whose body no reflection warnings are
issued?

Bye,
Tassilo
--
(What the world needs (I think) is not
(a Lisp (with fewer parentheses))
but (an English (with more.)))
Brian Hayes, http://tinyurl.com/3y9l2kf

Marshall T. Vandegrift

unread,
Nov 9, 2011, 6:47:54 AM11/9/11
to clo...@googlegroups.com
Tassilo Horn <tas...@member.fsf.org> writes:

> I'm facing the same issue. I have this macro for java interop:
>

> (defmacro with-traversal-context
> [[g tc] & body]
> `(let [old-tc# (.getTraversalContext ^Graph ~g)]
> (try
> (.setTraversalContext ^Graph ~g ^TraversalContext ~tc)
> ~@body
> (finally (.setTraversalContext ^Graph ~g ^TraversalContext old-tc#)))))
>

> But the type hints are gone in the macro expansion, thus I have 3
> reflection warnings per macro application, and real, performance
> critical reflection warnings get lost in the shuffle.

What you appear to be having is actually the different, but
conceptually-related problem of the metadata reader macro and unquote
operations interacting in a way which is consistent, but potentially not
optimal. Fortunately in your sort of situation, you can work around the
problem by replacing the metadata reader macro with explicit metadata
operations. Something like the following should work:

(defn assoc-meta [x & kvs]
(with-meta x (apply assoc (meta x) kvs)))

(defmacro with-traversal-context
[[g tc] & body]

(let [g (assoc-meta g :tag Graph)
tc (assoc-meta tc :tag TraversalContext)]
`(let [^TraversalContext old-tc# (.getTraversalContext ~g)]
(try
(.setTraversalContext ~g ~tc)
~@body
(finally (.setTraversalContext ~g old-tc#))))))

-Marshall

Tassilo Horn

unread,
Nov 9, 2011, 7:57:57 AM11/9/11
to clo...@googlegroups.com
"Marshall T. Vandegrift" <lla...@gmail.com> writes:

Hi Marshall,

>> I'm facing the same issue. I have this macro for java interop:
>>
>> (defmacro with-traversal-context
>> [[g tc] & body]
>> `(let [old-tc# (.getTraversalContext ^Graph ~g)]
>> (try
>> (.setTraversalContext ^Graph ~g ^TraversalContext ~tc)
>> ~@body
>> (finally (.setTraversalContext ^Graph ~g ^TraversalContext old-tc#)))))
>>
>> But the type hints are gone in the macro expansion, thus I have 3
>> reflection warnings per macro application, and real, performance
>> critical reflection warnings get lost in the shuffle.
>
> What you appear to be having is actually the different, but
> conceptually-related problem of the metadata reader macro and unquote
> operations interacting in a way which is consistent, but potentially
> not optimal. Fortunately in your sort of situation, you can work
> around the problem by replacing the metadata reader macro with
> explicit metadata operations.

Yes, that would probably do the trick. But on IRC, Alan already gave me
this recipe (gensyming the given parameter g) which is even a bit
shorter.

--8<---------------cut here---------------start------------->8---


(defmacro with-traversal-context
[[g tc] & body]

`(let [^Graph g# ~g
^TraversalContext old-tc# (.getTraversalContext g#)]
(try
(.setTraversalContext g# ~tc)
~@body
(finally (.setTraversalContext g# old-tc#)))))
--8<---------------cut here---------------end--------------->8---

--

Reply all
Reply to author
Forward
0 new messages