AOT/gen-class docs

[Rich Hickey]

I've started documenting AOT compilation and the new :gen-class option
for ns:

http://clojure.org/compilation

It's still a work in progress. Feedback welcome.

Rich

[Chas Emerick]

Looks good so far, Rich. Should be a blissfully smooth transition
from the "legacy" gen-class impl.

This is only tangentially related to the docs you're writing, but I
won't let that stop me:

As you know, I have at least one use case where being able to generate
gen-class specs from a macro (or, the full ns declaration, in the
absence of an evaluated gen-class spec). Going one step further than
that though, I'm wondering how best to approach contexts where one
would like to generate classes whose names don't match up with the
name of the current file (or with the package the generated class will
land from the current file). Right off the top of my head, easily and
cleanly generating BeanInfo classes based on attributes/methods/
interfaces/whatever of the current file's class would be super-handy.

This doesn't look possible right now -- adding a second, different ns
declaration does not generate the corresponding classfiles. The docs
you've written so far make it sound like this should work -- "The unit
of compilation is the namespace" -- but right now, it seems like files
are the unit of compilation. Just as an example, assume the following
is in com/foo/test.clj:

(ns com.foo.test
(:gen-class
:implements [java.util.Iterator]
:init ctor
:constructors {[String] []}
:state state))

(ns com.foo.test2
(:gen-class
:implements [java.util.Iterator]
:init ctor
:constructors {[String] []}
:state state))
---------

(compile 'com.foo.test) yields the com.foo.test classfiles one would
expect. However, (compile 'com.foo.test2) throws an exception, as
it's looking for a com/foo/test2.clj file (rather than using whatever
has already been defined in the com.foo.test2 namespace). I see why
that's the case, as compilation and loading are very tightly bound.
If that's going to remain the case, then I'd say that clj files are
the unit of compilation, not namespaces. It's not yet clear (to me,
anyway) whether the current state is an intermediate stable step, or
if the first cut on the docs are slightly off on this point.

- Chas

[Rich Hickey]

The unit of compilation will probably be called the 'lib', but that
isn't defined anywhere on the site yet. As far as 2 or more ns calls
in the same file, that violates the ns/file classpath requirements,
which say that my.ns.lib must be defined in my/ns/lib.clj. I'll make
the requirement that ns be the first expression in any file that
contains it explicit.

That doesn't make the unit of compilation the file though, as a
namespace can be defined in more than one file.

All that notwithstanding, I understand your needs and those of the few
others who have chimed in that were using gen-and-save-class, and I
hope to facilitate most of them soon.

One thing I have been considering is the possibility to opt-out of
generating a class when compiling (ns ...) via something like (ns
(:gen-class :external true) ...), which would state that the class
signature was going to be defined in a subsequent stand-alone gen-
class call. That way you could generate such calls via custom macros.

gen-class will always create a relationship between the stub class and
the load (__init) class, as well as the namespace in which the
implementations will be found, the naming conventions for matching
etc. It's a high-level feature with a lot of power, but by no means
represents a universal solution for all possible code generation
scenarios. I think there will be other applications of programmatic
class generation that will fall out of its scope, and people are free
to build whatever they want - if things prove generally useful they
can propose for core. Everyone can leverage the fact that ASM is built
in, and use gen-class as an example.

Rich

[Michael Reid]

> gen-class will always create a relationship between the stub class and
> the load (__init) class, as well as the namespace in which the
> implementations will be found, the naming conventions for matching
> etc. It's a high-level feature with a lot of power, but by no means
> represents a universal solution for all possible code generation
> scenarios. I think there will be other applications of programmatic
> class generation that will fall out of its scope, and people are free
> to build whatever they want - if things prove generally useful they
> can propose for core. Everyone can leverage the fact that ASM is built
> in, and use gen-class as an example.
>

Let me see if I follow the plan for gen-class:

By and large, most common cases will be handled with the :gen-class
option of (ns ...).

For those that need to generate classes in a more flexible manner, the
more primitive gen-class function (as opposed to gen-and-load-class or
gen-and-save-class) will still be exposed to users who need/want to
make use of that?

e.g. Chas' idea for generating BeanInfo classes would fall under a
call to gen-class, which has no constraints on the name of the class
generated in relation to the namespace.

And on top of that, the ASM lib is built into the distribution so
anyone who wants to generate fully customized bytecode can do so.

Do I have that right?

/mike.

[Rich Hickey]

On Nov 27, 11:42 am, "Michael Reid" <kid.me...@gmail.com> wrote:
> > gen-class will always create a relationship between the stub class and
> > the load (__init) class, as well as the namespace in which the
> > implementations will be found, the naming conventions for matching
> > etc. It's a high-level feature with a lot of power, but by no means
> > represents a universal solution for all possible code generation
> > scenarios. I think there will be other applications of programmatic
> > class generation that will fall out of its scope, and people are free
> > to build whatever they want - if things prove generally useful they
> > can propose for core. Everyone can leverage the fact that ASM is built
> > in, and use gen-class as an example.
>
> Let me see if I follow the plan for gen-class:
>
> By and large, most common cases will be handled with the :gen-class
> option of (ns ...).

Yes.

>
> For those that need to generate classes in a more flexible manner, the
> more primitive gen-class function (as opposed to gen-and-load-class or
> gen-and-save-class) will still be exposed to users who need/want to
> make use of that?
>

On the drawing board, so maybe.

> e.g. Chas' idea for generating BeanInfo classes would fall under a
> call to gen-class, which has no constraints on the name of the class
> generated in relation to the namespace.
>

As noted in what you quoted, there will be relationships between gen-
class classes, their loaders, a namespace, and the names of functions
implementing methods. I haven't decide yet how much of that I'm going
to parameterize.

> And on top of that, the ASM lib is built into the distribution so
> anyone who wants to generate fully customized bytecode can do so.
>

Yes, always has been.

Rich

[Rich Hickey]

I've made substantial enhancements to gen-class over the weekend.
Please refresh and read:

http://clojure.org/compilation

In short, I've made it so that you can call gen-class stand-alone.
This will generate a stub class at AOT compile time.

The relationship between a generated class and its implementing
namespace is now parameterized, as is whether or not the implementing
ns gets loaded by the class, and the prefix used by the method->var
mapping.

You can still use :gen-class in a ns directive, and many parameters
will be defaulted. (:gen-class) by itself will give you a class with
the same name as the ns, a main function, and "-" as the prefix. Note
that now, without any (:gen-class) in ns, you will _not_ get a stub
class generated.

This brings tremendous flexibility - you can define macros that expand
into gen-class calls, and both declare and implement multiple classes
in a single file.

I've also added gen-interface, based on a contribution from Chouser,
which works similarly at AOT compile time.

I've enhanced proxy so that it, too, will pre-generate the proxy
classes at AOT compile time, although doing so is not necessary in
order to use proxy. The interface of the proxy API is unchanged. Two
advantages come from this:

Proxies now have deterministic names, and so their construction can
be a direct constructor call, rather than reflective, as it was
previously.

Proxy generation was the last runtime code-gen/classloader
requirement. So the path is clear for building Clojure apps without
runtime codegen, for delivery in those environments that preclude it
(e.g. Android, unsigned applets). Looking forward to feedback from
people trying to reach those targets.

Rich

[Chouser]

On Tue, Dec 2, 2008 at 10:24 AM, Rich Hickey <richh...@gmail.com> wrote:
>
> Proxy generation was the last runtime code-gen/classloader
> requirement. So the path is clear for building Clojure apps without
> runtime codegen, for delivery in those environments that preclude it
> (e.g. Android, unsigned applets). Looking forward to feedback from
> people trying to reach those targets.

You asked for it. :-)

Here's a minimal applet .clj:

(ns net.n01se.Tree
(:gen-class
:extends java.applet.Applet))

Since it's missing a main fn, I would expect an exception like the this:
java.lang.UnsupportedOperationException: net.n01se.Tree/-main not
defined (NO_SOURCE_FILE:0)

Using svn 1136 I can compile and get the above exception from a normal
Clojure REPL, but if I try to use it as an applet:

$ appletviewer test.html
java.lang.ExceptionInInitializerError
at clojure.lang.Namespace.<init>(Namespace.java:31)
at clojure.lang.Namespace.findOrCreate(Namespace.java:116)
at clojure.lang.Var.internPrivate(Var.java:95)
at net.n01se.Tree.<clinit>(Unknown Source)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at java.lang.Class.newInstance0(Class.java:372)
at java.lang.Class.newInstance(Class.java:325)
at sun.applet.AppletPanel.createApplet(AppletPanel.java:798)
at sun.applet.AppletPanel.runLoader(AppletPanel.java:727)
at sun.applet.AppletPanel.run(AppletPanel.java:380)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.security.AccessControlException: access denied
(java.lang.RuntimePermission createClassLoader)
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:342)
at java.security.AccessController.checkPermission(AccessController.java:553)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.SecurityManager.checkCreateClassLoader(SecurityManager.java:611)
at java.lang.ClassLoader.<init>(ClassLoader.java:218)
at java.security.SecureClassLoader.<init>(SecureClassLoader.java:71)
at java.net.URLClassLoader.<init>(URLClassLoader.java:99)
at clojure.lang.DynamicClassLoader.<init>(DynamicClassLoader.java:30)
at clojure.lang.RT.<clinit>(RT.java:243)
... 14 more

You can see it found my class okay, but it looks like there may be
some dynamic classloader stuff still going on?

--Chouser

[Rich Hickey]

On Dec 2, 12:12 pm, Chouser <chou...@gmail.com> wrote:

> On Tue, Dec 2, 2008 at 10:24 AM, Rich Hickey <richhic...@gmail.com> wrote:
>
> > Proxy generation was the last runtime code-gen/classloader
> > requirement. So the path is clear for building Clojure apps without
> > runtime codegen, for delivery in those environments that preclude it
> > (e.g. Android, unsigned applets). Looking forward to feedback from
> > people trying to reach those targets.
>
> You asked for it. :-)
>
> Here's a minimal applet .clj:
>
> (ns net.n01se.Tree
> (:gen-class
> :extends java.applet.Applet))
>
> Since it's missing a main fn, I would expect an exception like the this:
> java.lang.UnsupportedOperationException: net.n01se.Tree/-main not
> defined (NO_SOURCE_FILE:0)

When and why?

The mappings are dynamic, you'll get an error if you try to call it,
and you could define that later somehow.

>
> Using svn 1136 I can compile and get the above exception from a normal
> Clojure REPL, but if I try to use it as an applet:
>
> $ appletviewer test.html
> java.lang.ExceptionInInitializerError
> at clojure.lang.Namespace.<init>(Namespace.java:31)
> at clojure.lang.Namespace.findOrCreate(Namespace.java:116)
> at clojure.lang.Var.internPrivate(Var.java:95)
> at net.n01se.Tree.<clinit>(Unknown Source)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
> at java.lang.Class.newInstance0(Class.java:372)
> at java.lang.Class.newInstance(Class.java:325)
> at sun.applet.AppletPanel.createApplet(AppletPanel.java:798)
> at sun.applet.AppletPanel.runLoader(AppletPanel.java:727)
> at sun.applet.AppletPanel.run(AppletPanel.java:380)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.security.AccessControlException: access denied
> (java.lang.RuntimePermission createClassLoader)

> You can see it found my class okay, but it looks like there may be
> some dynamic classloader stuff still going on?
>

Thanks, that's useful.

Yes, the dynamic loader is still there, just won't be used, but it
looks like the security check is on creation. Getting it completely
out is another project...

Rich