Arbitrary subexpressions with custom compiler flags

81 views
Skip to first unread message

Andy Fingerhut

unread,
Mar 20, 2012, 11:47:05 AM3/20/12
to cloju...@googlegroups.com
This is a more general version of the idea I mentioned earlier in the "Proposl for *warn-on-reflection* to be true by default" discussion. If there is interest, I can try creating a wiki page on dev.clojure.org and maybe even a patch.

The proposal there was a (known-reflection ...) form added to the compiler that was identical to the "do" form in all ways, except that the compiler sets *warn-on-reflection* to true within its lexical scope.

The new proposal is to extend "do" itself, or perhaps some new form, with the ability to set arbitrary compiler flags to arbitrary values within its lexical scope. For example, *unchecked-math*, *warn-on-reflection*, plus the other ones I might be forgetting right now, plus any new ones added in the future. I'd like to encourage your suggestions for what the syntax should look like, but an initial not-carefully-thought-out suggestion is something like:

(do {*warn-on-reflection* true, *unchecked-math* false}
;; normal do forms go here
)

Andy

Aaron Cohen

unread,
Mar 20, 2012, 2:03:49 PM3/20/12
to cloju...@googlegroups.com

^:unchecked-math (+ 1 2)

^:warn-on-reflection ^:unchecked-math
(do
(+ 1 2))

^{:warn-on-reflection false} ;; Suppresses reflection warnings even if
globally turned on
(do
(.method o))

Reads pretty well to me. It would be really easy to propagate this in
my CinC compiler, not sure how hard it would be to do in the CinJ
compiler.

Implemented properly it could be attached to any form rather than
having to make duplicates of all the forms you might want to attach
flags to.

Stuart Sierra

unread,
Mar 23, 2012, 9:02:17 AM3/23/12
to cloju...@googlegroups.com
Syntactically, I like the read-time metadata as Aaron C proposed, since that is already the common way to pass flags to the compiler (e.g. type hints). You'd need some way to enable it on a whole file, maybe with metadata on the `ns` declaration.

This would be a pretty substantial change to the current compiler, I think.
-S

Andy Fingerhut

unread,
May 30, 2012, 2:48:51 AM5/30/12
to cloju...@googlegroups.com
I've aleady done a first-draft implementation of a different variation of this idea, with a single new special form I called compile-time-let. It is demonstrated briefly here:

http://dev.clojure.org/display/design/Overriding+compile-time+var+values+at+expression-level+granularity

That implementation seems pretty straightforward to me -- the introduction of one new special form, and nothing else in the compiler is affected. compile-time-let can be used inside of macros without problems. Anywhere you have a subexpression in your code that you want to modify some compile-time settings for, you just wrap it in a compile-time-let with the desired bindings and you are done. It just works. (Please point out any problems you see with the idea that I could be missing.)

Aaron, were you proposing that the metadata would only work when followed by a (do ...) subexpression? If so, that would help limit the number of changes required for an implementation I've been considering so far, and you can skip reading the long-winded stuff that follows.


If you meant to be able to put such metadata before an arbitrary Clojure subexpression, the only way I know of to do that involves changes sprinkled throughout Compiler.java. One way is to modify the implementation of every single parse() method in Compiler.java (there are 28 of them) in nearly identical ways. Each modified parse() method would check whether there is metadata on the form being parsed, and if so, and if the metdata map has a key :warn-on-reflection, :unchecked-math, or some other name that affects compile-time behavior, then parse() would pushThreadBindings() to change the values of those vars in the scope of that expression, and pop those bindings just before returning from parse().

Can anyone think of a different way to implement this with the metadata syntax? I haven't thought of one, but I'm not very familiar with the compiler internals.


Assuming the "modify all parse() methods" is reasonable, I've put some debug print statements into a local copy of the compiler to see if the metadata is attached to the code objects being parsed. It turns out that it is, in many cases. For example InvokeExpr.parse() is called for the (conj x 2) subexpression of the following function definition, with a form object that has the desired metadata {:warn-on-reflection false} attached to it:

(defn foo [x] ^{:warn-on-reflection false} (conj x 2))

However, there are many other subexpressions that do not have metadata on them, and it seems to correspond to the cases when that subexpression is a macro. Examples:

;; let expands into let* with no metadata
(defn foo1 [x] ^{:warn-on-reflection false} (let [y 2] [x y]))

;; case expands into let* with no metadata
(defn foo1 [x]
^{:warn-on-reflection false}
(case x
0 (printf "zero")
1 (printf "one")))

Metadata is also not preserved for (loop ...) subexpressions, nor even a subexpression like (+ x 2), because the compiler transforms it into (. clojure.lang.Numbers (add x 2)) without the metadata that was on the (+ x 2) expression.

I also don't see metadata when I define my own macro, and then annotate a call to the macro with metadata.

Is it reasonable to consider making all macro expansions preserve metadata that was attached to the original, pre-expansion, expression? Is that possible?

Thanks,
Andy

Alan Malloy

unread,
May 30, 2012, 3:16:31 PM5/30/12
to Clojure Dev
On May 29, 11:48 pm, Andy Fingerhut <andy.finger...@gmail.com> wrote:
> However, there are many other subexpressions that do not have metadata on them, and it seems to correspond to the cases when that subexpression is a macro.  Examples:
>
> ;; let expands into let* with no metadata
> (defn foo1 [x] ^{:warn-on-reflection false} (let [y 2] [x y]))
>
> ;; case expands into let* with no metadata
> (defn foo1 [x]
>   ^{:warn-on-reflection false}
>   (case x
>     0 (printf "zero")
>     1 (printf "one")))
>
> Metadata is also not preserved for (loop ...) subexpressions, nor even a subexpression like (+ x 2), because the compiler transforms it into (. clojure.lang.Numbers (add x 2)) without the metadata that was on the (+ x 2) expression.
>
> I also don't see metadata when I define my own macro, and then annotate a call to the macro with metadata.
>
> Is it reasonable to consider making all macro expansions preserve metadata that was attached to the original, pre-expansion, expression?  Is that possible?

I have an eight-month-old patch that does this already:
http://dev.clojure.org/jira/browse/CLJ-865 - if you can spur it
forward to actually getting accepted, that would be lovely.

Andy Fingerhut

unread,
May 30, 2012, 7:21:30 PM5/30/12
to cloju...@googlegroups.com
Alan, thanks for the heads up on that ticket. I've seen it before many times while maintaining my list of prescreened patches, but had not yet looked at the issue in detail. I will try out your patch and see if it works as advertised.

The email discussion you linked to in the CLJ-865 description does make a suggestion that there might be a few macros where the author would like control over the metadata attached to the expansion, overriding the default your patch provides. If we assume the macro author ought to always "win" when they go to such trouble, it seems like "reserving" a new keyword for Clojure's use (like :tag, :line, and :file are now) where if its value were true in the macro-expansion's metadata it meant "I as the macro author want to ignore any user-supplied metadata and specify exactly what the macro-expansion's metadata will be", that seems like it would cover that hypothetical use case.

As far as spurring the ticket forward, you've given me an idea that perhaps I should sell highly-ranked slots early in the list of CLJ prescreened tickets :-) Seriously, I have made a little bit of a soapbox for myself in a few of the patch lists when there was a call asking if any patches ought to go in before the 1.4.0 release. Except for that, I'd rather not use the patch list for that purpose -- I like it better as "here is the state of things right now -- you decide what to do". I voted for CLJ-865, and can add a comment to it if I find it works as desired. I'm not sure what methods lie between that and buying beers for Clojure screeners (I wonder if that works... :-).

Andy

Alan Malloy

unread,
May 30, 2012, 9:08:47 PM5/30/12
to Clojure Dev
On May 30, 4:21 pm, Andy Fingerhut <andy.finger...@gmail.com> wrote:
> As far as spurring the ticket forward, you've given me an idea that perhaps I should sell highly-ranked slots early in the list of CLJ prescreened tickets :-)  Seriously, I have made a little bit of a soapbox for myself in a few of the patch lists when there was a call asking if any patches ought to go in before the 1.4.0 release.  Except for that, I'd rather not use the patch list for that purpose -- I like it better as "here is the state of things right now -- you decide what to do".  I voted for CLJ-865, and can add a comment to it if I find it works as desired.  I'm not sure what methods lie between that and buying beers for Clojure screeners (I wonder if that works... :-).

I actually hadn't even noticed who I was responding to when I said
that: I wouldn't suggest that you use your position of mild prominence
for politicking; I just meant that if my patch helped this other
desired feature (temporarily setting warn-on-reflection), that might
generate some more interest in it.

I like your idea of a special new flag saying "yes, I looked at the
caller's &form metadata, and here is what I'd like to expand to
anyway, thank you very much". I think it gets us a good default, while
still leaving ultimate power in the macro writer's hands. I'll add
that feature to the patch, and then try to get Chouser's attention
again and see if it passes muster.

Andy Fingerhut

unread,
Jun 4, 2012, 9:08:47 AM6/4/12
to cloju...@googlegroups.com
Thanks to helpful suggestions from Aaron Cohen, there is now a proof-of-concept patch implementing the metadata syntax idea for changing *warn-on-reflection* and *unchecked-math* on a per-subexpression granularity, attached as metadata-patch1.txt on this page:

http://dev.clojure.org/display/design/Overriding+compile-time+var+values+at+expression-level+granularity

There is also a new "Proposal B" section there with examples of the metadata syntax, copied from and/or inspired by Aaron's earlier email proposing it.

The proposed implementation only modifies compiler methods analyze and eval, so it is fairly small compared to my earlier thoughts on how to do it.

Andy

Aaron Cohen

unread,
Jun 4, 2012, 9:32:23 AM6/4/12
to cloju...@googlegroups.com
Andy, thanks for following this through. I like it.

Is there a ticket I can vote on (and do votes on tickets actually get
taken in to consideration?)

--Aaron
> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>

Andy Fingerhut

unread,
Jun 4, 2012, 10:01:15 AM6/4/12
to cloju...@googlegroups.com
There is no ticket yet. I recall Rich asking a couple of months ago for enhancements to go through a dev page first rather than creating a ticket right away. I thought I would try that here, since I wasn't sure whether there would be any interest in such a change.

I think the change would be useful even if it doesn't include *compiler-options* keys like :elide-meta and :disable-locals-clearing, but those could be added if there is interest. May as well get them all in one commit unless those add controversy.

Andy
Reply all
Reply to author
Forward
0 new messages