Binary compatibility and a culture of AOT abuse

[dysinger]

So in my experiments with using clojure / contrib w/ the "new" branch,
I've noticed a pattern of binary incompatibility. Jars pushed to
clojars, maven repos and other places that are are unnecessarily
AOTed, don't work with clojure "new". I noticed the same thing going
from clojure 1.0 to clojure 1.1.0-SNAP.

It doesn't really seem to matter if you include the src "clj" files
along with the class files in your jar. As soon as clojure hits a
classfile with an older AOTed namespace, it blows up. This isn't a
criticism of clojure. Clojure needs it's freedom to innovate.

...but clojure, by default, JIT compile clj files. I would just like
to start a conversation about using AOT sparingly if you publish your
clojure project as a library. Ask yourself "why am I AOTing this
namespace? Do I think JIT compiled dynamic clojure is dumb? Is this a
gen-class I need outside clojure in some fancy XML config file? Is
there some classloader issue that makes it necessary to compile AOT ?"
If the answer is NO, then please don't AOT the code.

I went through the pain of 3 days of on/off twiddling with libraries
so our project could use clojure "new". The hair-pulling details
where in all the code that was AOT when there was no reason to do
this. 99% of time there were no gen-class or anything else to require
it. People just like compiling stuff [ I get the feeling they think it
has some sort of ricer benefit to AOT compile ]. Most clj files work
great with all versions of clojure if you just jar the clj files and
leave class generation for runtime.

Don't get me wrong, there are places where a single namespace or two
are necessarily AOTed for interop or for classloader issues. But just
carte blanche AOT makes it harder on other people using your library -
to the point of trying to remove your library from their dependencies
(trust me - been there).

Code that _could_ run perfectly on clojure 1.0, 1.1.0-SNAP and "new"
are restricted to one version when AOTed and 99% for no reason.

Maybe Rich can comment on where I am missing it. But I think this is
my stance.

-Tim

[mac]

I've never really used many clojure libs at the same time or even any
very big ones so maybe I'm mistaken here, but does not AOT compilation
help startup time?
I know everyone is all internetty and webby these days but I kinda
like desktop apps myself and those need to start and stop a lot more
than server apps, which makes such things important.
Also from a developer point of view working with server software it's
nice if you can restart things quickly when the system has gotten into
a bad state because you carelessly deployed something broken on your
dev machine.

But if the time difference is not noticeable, then I certainly agree
with you for open source projects.

/Mac

[B Smith-Mannschott]

Yes, it does help considerably with startup time. I got into the habit
of AOT compiling my stuff because my primary development machine has a
1.6GHz Atom and starting up from "slim" builds was too slow for my
taste.

That said, I think Tim's point is well taken. I should back away from
AOT compiling again and see if the performance is still as much of a
problem as I remember.

This issue has also been brought up in connection with leiningen,
which currently AOT-compiles everything as part of its normal build.
My impression is that it would be smarter to be selective about what
gets AOT compiled, and what doesn't.

// Ben

[Philipp Meier]

Hi,

I like to suggest the following policy: publish libraries as jarred-
up .clj sources and AOT them locally when building an application.
Leiningen could sure help here and everen cache different compiled
versions dependening on the clojure version.

IIRC this is what Debian does with elisp packages.

Philipp Meier

[Laurent PETIT]

Hello,

Interesting return of experience.

But from what I (currently) know, AOTing jars should be harmless.
So maybe the problem is that there's a bug in the new branch, and this bug needs to be corrected ?

2009/12/13 dysinger <t...@dysinger.net>

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Chas Emerick]

There are certainly binary incompatibility issues between the
different versions/branches -- that'll settle out as the core matures,
and IIUC, especially once c-in-c becomes a reality.

However, only providing libraries as source (non-AOT-compiled) jars
whenever possible only shifts the problem around. Interop
requirements are often known of ahead of time (ha!), but many
"classloader issues" are totally in the hands of whomever is deploying
the end application, their java security policy, etc. In those
contexts, having only clojure source libraries available in maven
repos is an active hurdle to usage perhaps as irritating as having to
unwind "unnecessary" AOT compilation. Of course, people with a strict
AOT-compilation requirement can do the AOT-compilation on their own,
but I for one do not want to see Gentoo-esque norms emerge in the
Clojure community.

A more general solution would be to encourage library authors to
produce and deploy both source *and* binary, AOT-compiled artifacts,
and leave the choice to the library's users. I set this up on
Saturday for clutch (a clojure couchdb library and view server) in the
process of mavenizing the project:

http://github.com/cemerick/clutch/commit/aef461a6fd3c4797aef3b110677197f7351bb831

Then, to depend upon only the source jar in a downstream project, one
would simply add a 'classifier' element to the dependency element:

<dependency>
<groupId>com.ashafa</groupId>
<artifactId>clutch</artifactId>
<version>1.0-SNAPSHOT</version>
<classifier>sources</classifier>
</dependency>

I think offering both types of artifacts is a much more user-friendly
approach, is *very* easy to do (assuming maven usage), and resolves
all of the issues in play here (modulo the clojure.lang binary
signature incompatibilities that exist, but those should resolve
themselves in due course).

Cheers,

- Chas

[B Smith-Mannschott]

Hmmm... but in the Clojure world there's a third variant, isn't there?
i.e. something like 'slim' which contains AOT only where required:
namely in the case of genclass or such.

[Meikel Brandmeyer]

Hi Laurent,

Am 14.12.2009 um 09:43 schrieb Laurent PETIT:

> But from what I (currently) know, AOTing jars should be harmless.
> So maybe the problem is that there's a bug in the new branch, and this bug needs to be corrected ?

No. The inner workings of Clojure might change. AOT compiled code doesn't know these changes. I had this with some change to defmulti a while ago. However the source is always recompiled when it's loaded. So it always see the change and all is fine. (Of course only, as long as the change doesn't involve a public API.)

Sincerely
Meikel

[Laurent PETIT]

2009/12/14 Meikel Brandmeyer <m...@kotka.de>

So i's a kind of break of compatibility with AOT compiled code. That should be noted in future official clojure releases, indeed, since it requires every library to be recompiled.

[Chas Emerick]

On Dec 14, 2009, at 7:16 AM, B Smith-Mannschott wrote:

>> Then, to depend upon only the source jar in a downstream project, one
>> would simply add a 'classifier' element to the dependency element:
>>
>> <dependency>
>> <groupId>com.ashafa</groupId>
>> <artifactId>clutch</artifactId>
>> <version>1.0-SNAPSHOT</version>
>> <classifier>sources</classifier>
>> </dependency>
>
> Hmmm... but in the Clojure world there's a third variant, isn't there?
> i.e. something like 'slim' which contains AOT only where required:
> namely in the case of genclass or such.

Yes, that's true. This is something that lein and clojure-maven-
plugin et al. will have to consider going forward.

- Chas

[Phil Hagelberg]

B Smith-Mannschott <bsmit...@gmail.com> writes:

> This issue has also been brought up in connection with leiningen,
> which currently AOT-compiles everything as part of its normal build.
> My impression is that it would be smarter to be selective about what
> gets AOT compiled, and what doesn't.

Yep, this will be changed in the next release of leiningen. The default
build will only AOT namespaces that actually need it to
function. Naturally you will still be able to build jars that AOT
everything, but you will have to specifically ask for that behaviour.

-Phil

[Mark Derricutt]

On Tue, Dec 15, 2009 at 7:05 AM, Phil Hagelberg <ph...@hagelb.org> wrote:

> Yep, this will be changed in the next release of leiningen. The default
> build will only AOT namespaces that actually need it to
> function. Naturally you will still be able to build jars that AOT
> everything, but you will have to specifically ask for that behaviour.

I also have an almost working polyglot maven3 clojure module working
as well (reads and writes maven poms in clojure using a syntax
initially borrowed from leiningen but extended for the rest of mavens
bits and pieces - trying to sort out the syntax I want to use is
annoyingly hard!).

The way I was thinking of doing it was configuring up a default
assembly plugin instance to build the slim jar, a clojure:slimjar goal
on the clojure-maven-plugin ( probably the better place to put it ).