native libs in jars (using a custom classloader)

[ninjudd]

i want to make it easier to add jars that contain precompiled native
libraries to clojars. currently, to do this, you can use the lein
native-deps plugin, but users of your jar still have to know that it
has native dependencies, add them as :native-dependencies to
project.clj and run 'lein native_deps'. then they have to modify
LD_LIBRARY_PATH or java.libary.path by hand. this is much more
complicated than it should be. they should just have add the
dependency, type 'lein deps', and everything else should just work.

so, i've spent some time writing a custom classloader
(NativeClassLoader) that overrides findLibrary to locate and load
native library files transparently within jars. it works great, but in
the process of working on it, i ran into some code in the clojure
source that doesn't quite make sense to me.

i need to insert NativeClassLoader between
clojure.lang.DynamicClassLoader and sun.misc.Launcher$AppClassLoader.
i did this in the clojure code in two places:

1. clojure/main.clj at the top of the repl function
2. clojure/lang/RT.java inside of makeClassLoader

i guess i don't fully understand makeClassLoader and baseLoader.
makeClassLoader is called from Compile.java in three different places
(compile, eval and load). makeClassLoader wraps baseLoader with a
DynamicClassLoader, but it is happening multiple times. for example,
in master:

user=> (clojure.lang.RT/baseLoader)
#<DynamicClassLoader clojure.lang.DynamicClassLoader@70e35d5>
user=> (.getParent (clojure.lang.RT/baseLoader))
#<DynamicClassLoader clojure.lang.DynamicClassLoader@45b34126>
user=> (.getParent (.getParent (clojure.lang.RT/baseLoader)))
#<AppClassLoader sun.misc.Launcher$AppClassLoader@6ba7bf11>

the delegation hierarchy looks like this:
DynamicClassLoader -> DynamicClassLoader -> AppClassLoader

and when i insert NativeClassLoader, it looks like this:
DynamicClassLoader -> NativeClassLoader -> DynamicClassLoader ->
NativeClassLoader -> AppClassLoader

is this a bug? it seems like makeClassLoader should always wrap the
application classloader, not baseLoader, but there may be something
i'm missing. any help or advice would be appreciated. i'd like to open
source NativeClassLoader so everyone can benefit from it. is this
something others would find useful? Rich would you consider a patch to
clojure for this?

thanks,
Justin Balthrop

--
You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
To post to this group, send email to cloju...@googlegroups.com.
To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[Justin Balthrop]

I've got a working version of this now. It allows native JNI libraries
to be placed anywhere on the classpath, even inside jars. Native
libraries inside jars use the same directory structure as
leiningen.compile (e.g. native/macosx/x86_64/libjtokyocabinet.jnilib).

Here's the code:

NativeClassLoader.java:

http://gist.github.com/398896

Changes to clojure/main.clj and clojure/lang/RT.java:

http://gist.github.com/398903

I can't think of a way to insert my classloader without modifying the
clojure source though. I'd be fine with making this a standalone
package if I could figure out how. Maybe someone who knows more about
Java than I do has some ideas? I would have to figure out a way to set
the ContextClassLoader before Clojure does it. Is that possible?

-Justin

[Justin Balthrop]

So I think I figured out a way to do this without any modifications to
the clojure using *use-context-classloader*. The only problem is that
it doesn't work. Here's the code:

(defmacro import-native [& import-symbols-or-lists]
`(binding [*use-context-classloader* true]
(let [cl# (.getContextClassLoader (Thread/currentThread))]
(try
(.setContextClassLoader (Thread/currentThread)
(NativeClassLoader.))
(import ~@import-symbols-or-lists)
(finally
(.setContextClassLoader (Thread/currentThread) cl#))))))

I'm so close! The problem is that import doesn't use the context
classloader. It should, right? Or am I missing something?

-Justin

On May 12, 11:25 am, Justin Balthrop <goo...@justinbalthrop.com>
wrote:

[Justin Balthrop]

I fixed this in master: http://github.com/ninjudd/clojure/commit/4dac03fb2b47a33c77704c00fc9086f4c9b9dc3a

It looks to me like there are two bugs:

1. ImportExpr.emit is not consistent with ImportExpr.eval (http://
gist.github.com/402048)
ImportExpr.emit should generate bytecode that calls RT.classForName
rather than Class.forName. This way, RT.baseLoader is used rather than
using the ClassLoader of the caller, which is what Class.forName does.

2. *use-context-classloader* should default to false, as it did
originally, and it should always be obeyed by RT.baseLoader. (http://
gist.github.com/402056)
The current implementation only uses the context classloader when
Compile.LOADER isn't bound, which makes it impossible to use a custom
classloader for import.

Rich, I'd be happy to make a ticket for this if you make me a member
on Assembla. My username is ninjudd.

Thanks,
Justin

[Chas Emerick]

On May 15, 2010, at 2:19 AM, Justin Balthrop wrote:

> 1. ImportExpr.emit is not consistent with ImportExpr.eval (http://
> gist.github.com/402048)
> ImportExpr.emit should generate bytecode that calls RT.classForName
> rather than Class.forName. This way, RT.baseLoader is used rather than
> using the ClassLoader of the caller, which is what Class.forName does.
>
> 2. *use-context-classloader* should default to false, as it did
> originally, and it should always be obeyed by RT.baseLoader. (http://
> gist.github.com/402056)
> The current implementation only uses the context classloader when
> Compile.LOADER isn't bound, which makes it impossible to use a custom
> classloader for import.

This would be serious regression. Making that change is what enabled
clojure to be usable "out of the box" in environments that have more
"creative" classloader characteristics. In my experience, NetBeans
(for example) and various application servers absolutely required *use-
context-classloader* to be bound to true when loading clojure code, or
everything flew apart. Clojure is often loaded from a classloader
that has no visibility to application-level classloaders/resources, so
using the context classloader is key to ensuring reasonable default
functionality.

IMO, java.library.path configuration (and similar) is entirely a
tooling issue (in this case lein, or your editor/IDE). I'd
additionally wonder what the security implications might be for
automatically loading native libraries....not necessarily with the
knowledge of the end user, and by default from the (usually world-
writable) tmpdir?

That said, a clever spike for the problem. :-)

- Chas

[Justin Balthrop]

Chas, I'm downloading and installing NetBeans right now so I can
ensure any proposed changes work for that setup. In the meantime, I've
added debugging code to master and my fork so I can make sure that I
haven't changed which classloader is used when loading clojure code.
It looks like you're right that my initial change would break things
for environments with "creative" classloader characteristics.

I've modified the original gist to provide a patch that doesn't change
which classloader is used for loading code, but does fix the *use-
context-classloader* bug for import:
http://gist.github.com/402056

I think this change is closer to the original intention of the *use-
context-classloader* feature:
http://markmail.org/message/45v6osfx6cgnf6jk

And it fixes the problem addressed by the following commits more
directly:
http://github.com/richhickey/clojure/commit/b045a379215d1f48e6a2e6cedcdb41526cd5bb25
http://github.com/richhickey/clojure/commit/939976735ea071cb00944b066392ab8b8d749918

Aside from the merits of NativeClassLoader, *use-context-classloader*
should work correctly and allow developers to use a custom classloader
for import. That being said, I have considered the security
implications of dynamically extracting and loading native libraries
from jars, and I think changing the "java.io.tmpdir" system property
to a secure location on production machines should solve the issue you
raised.

Anyway, thanks for your response. I'll let you know once I get
NetBeans set up and can test these changes in that environment.

Cheers,
Justin

[Justin Balthrop]

Hey Chas,

I spent some time trying to get NetBeans to break with my patches, but
I couldn't reproduce the problem with either version of my patch. I'm
able to load and run clojure code just fine in NetBeans with my
modifications (and load dynamic libraries from Jars too). I was hoping
I could get it to fail with patch-v1 and work with patch-v2, thus
proving that I've addressed your concern.

I think I don't get any failures because the context classloader in
NetBeans is sun.misc.Launcher$AppClassLoader (the same as when
starting clojure from the command line). Any tips for reproducing the
problem in NetBeans or other environments I should test with? I'm
going to try it with Jetty when I get some more time.

-Justin


On May 15, 10:12 am, Justin Balthrop <goo...@justinbalthrop.com>
wrote:

> Chas, I'm downloading and installing NetBeans right now so I can
> ensure any proposed changes work for that setup. In the meantime, I've
> added debugging code to master and my fork so I can make sure that I
> haven't changed which classloader is used when loading clojure code.
> It looks like you're right that my initial change would break things
> for environments with "creative" classloader characteristics.
>
> I've modified the original gist to provide a patch that doesn't change
> which classloader is used for loading code, but does fix the *use-
> context-classloader* bug for import:http://gist.github.com/402056
>
> I think this change is closer to the original intention of the *use-
> context-classloader* feature:http://markmail.org/message/45v6osfx6cgnf6jk
>
> And it fixes the problem addressed by the following commits more

> directly:http://github.com/richhickey/clojure/commit/b045a379215d1f48e6a2e6ced...http://github.com/richhickey/clojure/commit/939976735ea071cb00944b066...

[Chas Emerick]

The NetBeans failures crop up when using clojure as the basis of a
NetBeans Module, which is then depended upon by another NetBeans
Module (as is commonplace when building NetBeans plugins). None of
this is relevant when simply using an enclojure REPL or somesuch, for
example. Sorry for not being specific before.

Looking back at the thread from 2008 you referenced, a lot of things
have changed since then, many of which I haven't followed (e.g. add-
classpath usage is strongly discouraged, and the scope of
DynamicClassloader usage has been trimmed back quite a bit last I knew
-- and IIRC, Rich has said he'd like to eliminate it entirely).
That's all to say that this may all be tilting at windmills if you're
swimming against the tide, as it were.

- Chas

[Justin Balthrop]

Well, I spent some time trying to reproduce your failure scenario in
NetBeans. Gives new meaning to the phrase "swimming against the tide".
Anyway, I couldn't get it to break the way you described, but it's
quite possible I haven't duplicated your scenario correctly. Chas, if
you get time, and you're feeling altruistic, you can download my
clojure branch at http://github.com/ninjudd/clojure and see if it
works.

Speaking of windmills, I think eliminating DynamicClassLoader would
probably fix the problem I'm seeing as I assume it would mean just
using the context classloader all the time.

-Justin