java.ext.dirs problem
Chouser
at a "classpaths" directory. That's where I dump jars,
symlinks to jars, and symlinks to the 'src' and 'classes'
directories of the various projects I have installed.
I know that some people have had trouble with java.ext.dirs
and instead use wildcards in their classpaths or other more
complicated solutions. But since I've never seen these
problems described in enough detail to reproduce or
troubleshoot, when I stumbled on one tonight I thought it'd
be worth documenting...
Here is the simplest way I've found to produce the error I'm
seeing:
1. Compile clojure-contrib
2. Create a "classpaths" directory with symlink to
clojure-contrib/classes, so as to add the compiled contrib
classes to the classpath. Mine lookes like this:
$ ls -l classpaths/
total 0
lrwxrwxrwx 1 chouser chouser 32 2009-06-27 00:54 classes ->
../proj/clojure-contrib/classes/
3. Start clojure and try to use something from contrib:
java -Djava.ext.dirs=classpaths -cp proj/clojure/clojure.jar clojure.main
user=> (require 'clojure.contrib.math)
java.lang.NoClassDefFoundError: clojure/lang/RT (NO_SOURCE_FILE:0)
The full stack trace for that exception is attached.
I've found two workarounds so far. Take your pick:
A. Add clojure.jar to the "classpaths" directory. Once
that's done, it doesn't matter if you also specify it in -cp
on the java command line or not:
$ ls -l classpaths/
total 0
lrwxrwxrwx 1 chouser chouser 32 2009-06-27 00:54 classes ->
../proj/clojure-contrib/classes/
lrwxrwxrwx 1 chouser chouser 27 2009-06-27 01:06 clojure.jar ->
../proj/clojure/clojure.jar
B. Remove the contrib link from "classpaths" and specify it
on the java command line instead:
java -Djava.ext.dirs=classpaths -cp proj/clojure-contrib/classes clojure.main
--Chouser
Laurent PETIT
know enough to say that placing your libs in the java.ext.dirs
classpath is a trick that could lead to problems.
It's primary and sole intent is to hold extensions to the java API
that then would be loaded with the core java APIs.
That's certainly why you're having problems when you mess up the
dependencies in places where there is a hierarchy of classloaders
involved : clojure-contrib was loaded with the "main classloader"
along with java core libraries, and clojure was loaded with a "child"
classloader. And thus clojure-contrib classes were not able to see any
of the classes in the child classloader.
The problem with this java.ext.dirs is of the category of problems
pointed out by C. Martin (and others!) as a design smell : the one
named "viscosity" : "doing things right is harder than doing things
wrong". It's so easy to put everything in this java.ext.dirs directory
that it's difficult resisting the temptation !
Regards,
--
Laurent
2009/6/27 Chouser <cho...@gmail.com>:
mccraigmccraig
configures java classpaths and invokes clojure [only tried on macos...
probably fine on linux, and should be portable to windows]
http://github.com/mccraigmccraig/clojure-load.rb
if you put all your clojure project repos, including clojure-load.rb,
in one directory you can invoke it so :
clojure -l clojure-json
clojure and clojure-contrib are loaded by default at the end of the
classpath. if a repo doesn't follow the conventional layout, or you
need more control, you can specify the classpath order and add dirs of
jars or individual jars or dirs. e.g.
clojure -l clojure -l clojure-contrib -p compojure/classes -p
compojure/src -j compojure/deps
and you can give arbitrary jvm args or script args without setting
environment vars, using the -- separator :
clojure --script -l clojure-json -- -Xmx1g -- script/path args
ruby is another dependency : i just couldn't face doing it in bash :)
c
Chouser
>
> While I'm far from a java classpath-related issues problem, I think I
> know enough to say that placing your libs in the java.ext.dirs
> classpath is a trick that could lead to problems.
I've been using it mainly because Rich has recommended it:
http://clojure-log.n01se.net/date/2008-10-26.html#15:33
http://clojure-log.n01se.net/date/2008-11-22.html#11:58
> It's primary and sole intent is to hold extensions to the java API
> that then would be loaded with the core java APIs.
Is there any documentation suggesting it should not be used
the way I'm using it? The Sun docs I've read so far don't
seem to forbid it.
> That's certainly why you're having problems when you mess up the
> dependencies in places where there is a hierarchy of classloaders
> involved : clojure-contrib was loaded with the "main classloader"
> along with java core libraries, and clojure was loaded with a "child"
> classloader. And thus clojure-contrib classes were not able to see any
> of the classes in the child classloader.
I see. But even this is not a problem when loading code
from .clj files instead of .class files, which may be why
I haven't run into it much.
...and actually doesn't it have to be a bit more complicated
than that? Otherwise I would have been unable to load *any*
classes named only in the java.ext.dirs via the -cp-loaded
clojure REPL, right?
Anyway, I have a solution (in complete violation of your
warnings!) that's working nicely now. :-) I have
a cp-common directory, as well as one or more cp-<proj>
dirs, where a <proj> can specify either just a clojure
branch (master, 1.0, chunks, or an application project I'm
working on. In either case, it can contain a link to the
specific clojure jar I want. For example, cp-1.0 can
include links to both clojure-1.0.jar and the 1.0 branch of
contrib (once it exists), while clojure-master can include
links to the lastest pull of each.
So is this still the wrong way? Is the right way really to
launch ruby so I can launch clojure!?
--Chouser
Laurent PETIT
2009/6/28 Chouser <cho...@gmail.com>:
>
> On Sat, Jun 27, 2009 at 2:34 AM, Laurent PETIT<lauren...@gmail.com> wrote:
>>
>> While I'm far from a java classpath-related issues problem, I think I
>> know enough to say that placing your libs in the java.ext.dirs
>> classpath is a trick that could lead to problems.
>
> I've been using it mainly because Rich has recommended it:
>
> http://clojure-log.n01se.net/date/2008-10-26.html#15:33
> http://clojure-log.n01se.net/date/2008-11-22.html#11:58
Well, I can't but agree to say that using java.ext.dirs is the
simplest thing to do.
It's just that placing one's own "global libs" there has always worked for now.
More a side effect than a feature, if you believe me (and clojure is
all about minimizing / isolating side effects :-).
>
>> It's primary and sole intent is to hold extensions to the java API
>> that then would be loaded with the core java APIs.
>
> Is there any documentation suggesting it should not be used
> the way I'm using it? The Sun docs I've read so far don't
> seem to forbid it.
So far, I remember that java documentations lead people to place
libraries that provide extensions of well identified java APIs in
java.ext.dir, and to set user / application libraries in the classpath
via -cp et al.
What I want to add is I'm not a specialist, but looking at rather
*big* projects such as Apache tomcat, Eclipse, ... resisting the
temptation to abuse the java.ext.dir directory for their own launch
scripts tends to infer there's some reason to do so. Just testing that
in the very basic case of one's own laptop it does not pose any
problem is not enough for recommanding this to anybody.
I understand well that *it will be ok* for the user while he is
testing the app on his machine, but by doing so, he is certainly far
from the configuration where the app will finally be deployed (think a
webapp server, a server where java.ext.dir is controlled by an admin,
...). This could lead (from time to time), to unexpected and annoying
surprises at deployment time ...
>
>> That's certainly why you're having problems when you mess up the
>> dependencies in places where there is a hierarchy of classloaders
>> involved : clojure-contrib was loaded with the "main classloader"
>> along with java core libraries, and clojure was loaded with a "child"
>> classloader. And thus clojure-contrib classes were not able to see any
>> of the classes in the child classloader.
>
> I see. But even this is not a problem when loading code
> from .clj files instead of .class files, which may be why
> I haven't run into it much.
Interesting problems, since they force to think hard about all those
classloader issues :-)
Indeed, since resource files (clj files) are probably delivered by the
same classloaders that would have delivered class files, it's weird
that it works in one case and not the other. Maybe I don't understand
the differences sufficiently precisely, after all :-$
>
> ...and actually doesn't it have to be a bit more complicated
> than that? Otherwise I would have been unable to load *any*
> classes named only in the java.ext.dirs via the -cp-loaded
> clojure REPL, right?
Isn't it the other way around ?
"
Caused by: java.lang.NoClassDefFoundError: clojure/lang/RT
at clojure.contrib.math__init.<clinit>(Unknown Source)
"
your exception shows that what you placed in java.ext.dir is unable to
use a class from clojure.jar since it is provided by a classloader
that is not a parent of the master classloader.
So you were able to launch everything from clojure since it was on the
user classpath, but the first time an already compiled class loaded
via the master classloader asks its classloader to launch something
from clojure.lang ... CLABAMGO !
> Anyway, I have a solution (in complete violation of your
> warnings!) that's working nicely now. :-) I have
> a cp-common directory, as well as one or more cp-<proj>
> dirs, where a <proj> can specify either just a clojure
> branch (master, 1.0, chunks, or an application project I'm
> working on. In either case, it can contain a link to the
> specific clojure jar I want. For example, cp-1.0 can
> include links to both clojure-1.0.jar and the 1.0 branch of
> contrib (once it exists), while clojure-master can include
> links to the lastest pull of each.
This seems good since dependencies of every project are under control.
Then having a shell script that will walk your deps to fill the -cp
arg or placing everything in java.ext.dirs is the implementation part,
but it seems to me you have the "specification" part right :-)
>
> So is this still the wrong way?
If I said something like "it is the wrong way", then I was probably in
a bad mood and not really myself, 'cause I generally don't think there
is just one way. But I guess (partly objectively, partly intuitively
<- this one being maybe the more important one :-) that it is a
dangerous/delicate way.
> Is the right way really to
> launch ruby so I can launch clojure!?
Generally, there's really no need to bring in ruby for that ! :-)
Projects I have seen so far (e.g. tomcat, ...) provide .sh and .bat
scripts and that's good enough.
Specifically, I remember there has been some posts about .sh scripts
with decent proposals for multiproject directory hierarchies ... ?
Personally, I'm not a sh guru either, and I start my projects in
clojuredev (shameless plug ! :-) where there is an ancestral ! java
development plugin which has solved all these issues years ago (back
in 2000 or before) by providing some click & play tabs for setting
dependencies !
Regards,
--
Laurent
Alex Combas
Undoubtedly it is not the best solution, but all I do is export a variable called CLASSPATH
pointing to whichever directories I want to require or load.
# append to your .bashrc script
# Start
CLASSPATH=/home/user/aaa:/home/user/bbb:$CLASSPATH
export CLASSPATH
# End
The only gotcha so far that I've discovered is that you must not include the namespace in the path.
For example with a file located here:
/home/user/project/foo/bar.clj
If the namespace of that file is (ns foo/bar) then do this:
# correct
CLASSPATH=/home/user/project:$CLASSPATH
export CLASSPATH
# wrong
CLASSPATH=/home/user/project/foo:$CLASSPATH
export CLASSPATH
As for distributing your project all you would need is to put something like this
into the script that you already use to launch your program.
This is pretty simple but it works for me, just update my CLASSPATH variable and I'm good to go
and I don't have to specify anything on the command line when I launch a repl.
Best regards,
agc
Christophe Grand
>> That's certainly why you're having problems when you mess up theInteresting problems, since they force to think hard about all those
>> dependencies in places where there is a hierarchy of classloaders
>> involved : clojure-contrib was loaded with the "main classloader"
>> along with java core libraries, and clojure was loaded with a "child"
>> classloader. And thus clojure-contrib classes were not able to see any
>> of the classes in the child classloader.
>
> I see. But even this is not a problem when loading code
> from .clj files instead of .class files, which may be why
> I haven't run into it much.
classloader issues :-)
Indeed, since resource files (clj files) are probably delivered by the
same classloaders that would have delivered class files, it's weird
that it works in one case and not the other. Maybe I don't understand
the differences sufficiently precisely, after all :-$
Except that resource files don't depend on clojure.core classes. (In truth a resource file can't have dependencies.)
My current understanding of the problem is that although sources on java.ext.dirs are loaded from the "root" clasloader, they are compiled and the resulting classes are loaded by a classloader which is a descendant of the classloader that loaded Clojure.
Laurent PETIT
Yes, that's certainly the missing part of the explanation, thanks !
In one case : clojure.core asks the root classloader to load already
compiled classes, which, in turn, from the root classloader, try to
use clojure.lang.RT the root class loader is unable to find.
In the other case : clojure.core asks the root classloader to load the
.clj files, and it is compiled from within clojure.core's classloader,
which is able to find everything inside clojure.jar.
Cheers !
--
Laurent