fn compile listener

[MikeM]

I've been experimenting with the following patches to
clojure.lang.Compiler and clojure.lang.RT. They allow a callback to a
clojure function when the compiler compiles a fn.

With the patches you can, for example:

(in-ns 'clojure)
(defn *fncompile-listener*)
[name form] (println name form))

(in-ns 'user)
(clojure/refer 'clojure)
(defn myfunc
[s] (map (fn[a] (* a 2)) s))

with the following output:

user.myfunc__4659$fn__4661 (fn* ([a] (* a 2)))
user.myfunc__4659 (fn* ([s] (map (fn [a] (* a 2)) s)))


The intent is to be able to capture the generated class names and
associated forms for reference when examining stack traces, profiling
information, and perhaps other uses.

Note that using the binding macro for *fncompile-listener* typically
won't give the desired result, since the inner forms are compiled
before the binding takes effect, but binding can be used in cases
where it is applied to a function that evals, eg

(binding [*fncompile-listener* (fn[a b] (println "[fncompile]" a b))]
(eval '(map (fn[a] (* a 2)) [1 2 3])))

gives the following:

user.eval__4779$fn__4781 (fn* ([a b] (println [fncompile] a b)))
user.eval__4779 (fn* ([] (binding [*fncompile-listener* (fn [a b]
(println [fncompile] a b))] (eval (quote (map (fn [a] (* a 2)) [1 2
3]))))))
[fncompile] user.eval__4785$fn__4787 (fn* ([a] (* a 2)))
[fncompile] user.eval__4785 (fn* ([] (map (fn [a] (* a 2)) [1 2 3])))

Although this binding example is a bit awkward, I'm able to use
binding with *fncompile-listener* in a clean way in my GUI REPL, since
I have an evaluator function that I can wrap easily.

Rich - please consider incorporating these patches (or a similar
capability). Thanks.



Index: C:/Documents and Settings/MESSINM2/My Documents/workspace/
clojure/src/jvm/clojure/lang/Compiler.java
===================================================================
--- C:/Documents and Settings/MESSINM2/My Documents/workspace/clojure/
src/jvm/clojure/lang/Compiler.java (revision 957)
+++ C:/Documents and Settings/MESSINM2/My Documents/workspace/clojure/
src/jvm/clojure/lang/Compiler.java (working copy)
@@ -192,6 +192,9 @@
//DynamicClassLoader
static final public Var LOADER = Var.create();

+//mm 23jul08 - experimental listener for fn compile
+static final public Var FNC_LISTENER =
Var.find(Symbol.create("clojure","*fncompile-listener*"));
+
enum C{
STATEMENT, //value ignored
EXPRESSION, //value required
@@ -2758,6 +2761,10 @@
Var.popThreadBindings();
}
fn.compile();
+ //mm 23jul08 experimental fn listener
+ if(FNC_LISTENER.get()!=null)
+ ((IFn)FNC_LISTENER).invoke(fn.name,form);
+
return fn;
}

Index: C:/Documents and Settings/MESSINM2/My Documents/workspace/
clojure/src/jvm/clojure/lang/RT.java
===================================================================
--- C:/Documents and Settings/MESSINM2/My Documents/workspace/clojure/
src/jvm/clojure/lang/RT.java (revision 957)
+++ C:/Documents and Settings/MESSINM2/My Documents/workspace/clojure/
src/jvm/clojure/lang/RT.java (working copy)
@@ -142,6 +142,7 @@
final static Var PRINT_READABLY = Var.intern(CLOJURE_NS,
Symbol.create("*print-readably*"), T);
final static Var WARN_ON_REFLECTION = Var.intern(CLOJURE_NS,
Symbol.create("*warn-on-reflection*"), F);

+final static Var FNC_LISTENER = Var.intern(CLOJURE_NS,
Symbol.create("*fncompile-listener*"), null);
//final static Var IMPORTS = Var.intern(CLOJURE_NS,
Symbol.create("*imports*"), DEFAULT_IMPORTS);
final static IFn inNamespace = new AFn(){
public Object invoke(Object arg1) throws Exception{

[Chas Emerick]

Pretty cool. Two comments:

- Don't we really want to listen to all def's, not just fn's? I say
this because this is moving towards enabling "plugins" for the Clojure
compiler (quoted because listening allows you to do things like
generate docs out the side of the compiler, but doesn't allow one to
modify or add to the compiler's output).

- We shouldn't assume that there's only one listener/plugin at a time;
perhaps instead of binding a listener var, a macro should be available
that conses a new listener onto a list of listeners at that var and
uses that as the new binding of the listeners' var. That doesn't
prevent someone from explicitly clobbering that list further down the
stack, but it would help make the common case safer for all.

Cheers,

- Chas

[MikeM]

Thanks for the interest and comments.

>
> - Don't we really want to listen to all def's, not just fn's?  I say
> this because this is moving towards enabling "plugins" for the Clojure
> compiler (quoted because listening allows you to do things like
> generate docs out the side of the compiler, but doesn't allow one to
> modify or add to the compiler's output).

I can see that other actions of the compiler could be of interest, and
the same approach could be used for any number of these actions. It
may become a little unwieldy if all compiler actions are listenable,
but perhaps there's a way to do this - maybe a defmulti to help sort
out the various calls that would result.

>
> - We shouldn't assume that there's only one listener/plugin at a time;
> perhaps instead of binding a listener var, a macro should be available
> that conses a new listener onto a list of listeners at that var and
> uses that as the new binding of the listeners' var.  That doesn't
> prevent someone from explicitly clobbering that list further down the
> stack, but it would help make the common case safer for all.
>

I think having one var is sufficient for the Compiler to interface to
- I think your idea could be implemented as a layer over the listener
var, although that leaves the possibility of clobbering as you say. I
lean towards keeping the Compiler listener interface simple and adding
on any additional facilities using clojure as needed.

[Rich Hickey]

It's an interesting idea. I agree with Chas, any listener system
should support multiple listeners, and care needs to be taken to make
sure it's thread safe.

Another issue is the fact that any such hook exposes implementation
details that, once exposed, become more difficult to change.

This latter point and the fact that I don't understand the use cases
all that well causes me to not want to do this at this time. I think a
more considered approach would somehow abstract the notion of
'compiler event' and supply a single stream, including warnings etc,
rather than such a specific hook point/callback.

I'll keep thinking about it,

Rich

[MikeM]

Ok, thanks for considering it. I understand the concerns you raise.

As for use cases, I thought of the idea when I had a hard time tracing
an exception back to an anonymous function. There are other ways to
address this problem, but it seemed that having the ability to see all
of the classes that get generated from clojure code and the associated
clojure forms would be generally useful. Although I haven't done much
profiling, I believe java profiling tools generally report results by
class, so having the ability to quickly review profiling results
against uncompiled forms could be helpful.