Clojure creates lots of classloaders

[Hendrik]

Hi,

I got a question: Clojure seems to create poopillions of
DynamicClassLoader instances. Why does it do that? Could I try
patching it so that it creates less of them? I need this cause I ran
into trouble working with JNI.

I looked at the Clojure source rev 1323. I'm no expert, this is my
first look at the sources, but the following struck me as hack-ish:
(from Compiler.java, shortened slightly)

public static Object eval(Object form) throws Exception{
if(true)//!LOADER.isBound())
makeClassLoader();

To me, that looks like for evaluating _every form_, a new classloader
gets created. Isn't that a bit much? Also, it looks "temporary" that
the if() has its condition commented out and replaced with
true ... :-/ So Clojure's behavior seems to have been different at
some point in the past.

Thanks a bunch for your opinions.

Cheers
Hendrik



Details:

I'm asking because I was loading a native library through JNI. The
rule here (apparently) is that you can load a library through only one
classloader, not the same library through several loaders.

For example, this program fails:

(System/load "/home/hk/inotify-java-0.1/build/native/libinotify-
java.so")
(System/load "/home/hk/inotify-java-0.1/build/native/libinotify-
java.so")

I feel that the program should work. Apparently Clojure creates new
classloaders between line 1 and 2. An equivalent Java program does
work (the first System.load() loads the library, the second sees the
library is already loaded and does nothing).

This is the exact error message:

$ cat t3.clj
(System/load "/home/hk/inotify-java-0.1/build/native/libinotify-
java.so")
(System/load "/home/hk/inotify-java-0.1/build/native/libinotify-
java.so")
(println "42")

$ java -jar /home/hk/clojure-svn-2/clojure.jar t3.clj
java.lang.UnsatisfiedLinkError: Native Library /home/hk/inotify-
java-0.1/build/native/libinotify-java.so already loaded in another
classloader (t3.clj:0)
at clojure.lang.Compiler.eval(Compiler.java:4533)
at clojure.lang.Compiler.load(Compiler.java:4846)
at clojure.lang.Compiler.loadFile(Compiler.java:4813)
at clojure.main$load_script__5685.invoke(main.clj:206)
at clojure.main$script_opt__5716.invoke(main.clj:258)
at clojure.main$main__5740$fn__5742.invoke(main.clj:333)
at clojure.main$main__5740.doInvoke(main.clj:328)
at clojure.lang.RestFn.invoke(RestFn.java:413)
at clojure.lang.Var.invoke(Var.java:346)
at clojure.lang.AFn.applyToHelper(AFn.java:173)
at clojure.lang.Var.applyTo(Var.java:463)
at clojure.main.main(main.java:39)
Caused by: java.lang.UnsatisfiedLinkError: Native Library /home/hk/
inotify-java-0.1/build/native/libinotify-java.so already loaded in
another classloader
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1743)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1674)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1005)
at user$eval__4.invoke(t3.clj:2)
at clojure.lang.Compiler.eval(Compiler.java:4522)
... 11 more

$

[Joshua Fox]

I think it is the best way to load and discard lots of dynamically-loaded and in fact dynamically-generated classes.

Joshua

[pmf]

Without dedicated classloaders, temporary (dynamically created)
classes would leak (since there is no way to unload a class without
letting its classloader be garbage collected). This might or might not
be the reason why Clojure uses many classloaders.

[Jeffrey Straszheim]

This was to fix a bug where new code (generated by (eval ...)) would never get garbage collected and crashed some programs that used eval heavily.  The JVM has a limitation that it will never GC a loaded class, but can GC a collection of classes referenced by a single classloader.

[Chas Emerick]

Hendrik,

I came across this issue with JNI libs some years ago, and also comes
up in the context of most app servers, which use a multitude of
classloaders to enable hot code reloading and such. The general
solution is to move your JNI lib into your boot classpath; in my
experience with Tomcat and Weblogic years ago, doing this caused the
boot classloader to link the JNI lib, making it available from all
child classloaders. See #3 here: http://www.bostic.com/docs/bdb_current/ref/java/faq.html

Creating a new classloader is really important to enable proper GC'ing
of any dynamically-generated classes later on, as others have said.
This also only happens when one loads code, which will generally only
happen at or near app startup outside of a development environment, so
performance isn't really a concern IMO.

- Chas

[Jeffrey Straszheim]

Many people consider the use of eval in "normal" code to be bad style.  However, there are times when it is justified, such as genetic programming or dynamic code loading.  In these cases Clojure falls down without the classloader trick.

[Chas Emerick]

That's a good point. Does such usage cause some failure, or is it a
performance issue? I would think that prior generations' code
(classes + classloader(s)) would get GC'd as necessary -- or is the
number of created classloaders so significant as to hit some serious
limitation?

- Chas

[Jeffrey Straszheim]

I dies with a MaxPermGen exception.