JNI & other non-reloadable JARs
Steven Parkes
It may be that things have progressed and I've missed them, but I still need a way to load JARs above anything that's going to get unloaded. Primarily for JNIs (sqlite), but also for some JDBC stuff. (The JDBC stuff could be worked around by some kind of callback for de-registering drivers when they're unloaded, but I don't see a solution for JNIs.)
Note that forking doesn't help this, I don't think. The JNI is getting loaded as needed during compilation and I don't think the forking wraps that (for obvious reasons).
The old way I hacked on this is gone, which is just as well. It was pathetic.
But I'm really not sure the best approach. The one thing I do know is that no approach isn't answer. I just don't have a choice: I can't restart sbt every time I run tests that use sqlite.
I don't really see any good solution, but I don't claim to understand all the subtitles of this. My hack would be to add a mutable field to ClasspathUtilities and prefer that over rootLoader if it's not nil. That's easy enough (I think) to bash when my project gets loaded, if it's not already set.
Not a good solution and requires me keep my own copy of sbt. But a sucky solution is better than no solution.
I had the vague idea that it might be possible to stuff this stuff into launcher's boot path which would probably be okay. But I'm not really sure.
So any suggestion would be gladly welcome ...
Mark Harrah
Steven Parkes <smpa...@smparkes.net> wrote:
> I'm looking at migrating from 0.7 to 0.10 � and I'd really like to get rid of my class loader hacks. Keeping my own copy of sbt around (and built and up to date) is painful �
>
> It may be that things have progressed and I've missed them, but I still need a way to load JARs above anything that's going to get unloaded. Primarily for JNIs (sqlite), but also for some JDBC stuff. (The JDBC stuff could be worked around by some kind of callback for de-registering drivers when they're unloaded, but I don't see a solution for JNIs.)
>
> Note that forking doesn't help this, I don't think. The JNI is getting loaded as needed during compilation and I don't think the forking wraps that (for obvious reasons).
It certainly would help for 'run', but that is the only thing that forks. Testing and web application running do not fork. I don't see why 'compile' need JNI to be loaded unless it is used by a compiler plugin.
> The old way I hacked on this is gone, which is just as well. It was pathetic.
>
> But I'm really not sure the best approach. The one thing I do know is that no approach isn't answer. I just don't have a choice: I can't restart sbt every time I run tests that use sqlite.
>
> I don't really see any good solution, but I don't claim to understand all the subtitles of this. My hack would be to add a mutable field to ClasspathUtilities and prefer that over rootLoader if it's not nil. That's easy enough (I think) to bash when my project gets loaded, if it's not already set.
>
> Not a good solution and requires me keep my own copy of sbt. But a sucky solution is better than no solution.
>
> I had the vague idea that it might be possible to stuff this stuff into launcher's boot path which would probably be okay. But I'm not really sure.
You can by modifying sbt.boot.properties for the launcher. See:
https://github.com/harrah/xsbt/wiki/Launcher
A better solution might be to use the new interface that makes it easier to add to sbt's classpath. See:
https://github.com/harrah/xsbt/wiki/Specialized
If it works, this could probably be turned into a proper plugin.
-Mark
Steven Parkes
Sorry. It's been a long day.
> Testing and web application running do not fork
Yeah, I confused test:run and test:test.
> A better solution might be to use the new interface that makes it easier to add to sbt's classpath. See:
>
> https://github.com/harrah/xsbt/wiki/Specialized
>
> If it works, this could probably be turned into a proper plugin.
That sounds a lot better, if I can figure out how to make it to work ...
Steven Parkes
>
> https://github.com/harrah/xsbt/wiki/Launcher
>
> A better solution might be to use the new interface that makes it easier to add to sbt's classpath. See:
>
> https://github.com/harrah/xsbt/wiki/Specialized
>
> If it works, this could probably be turned into a proper plugin.
Could be off base here … I'm not going to write "I think" after every statement, but you can insert it ...
To be general, these non-reloadable things have to be loaded above all the ScalaProviders, otherwise using a different version of Scala in the same session (if that's the right term) is going to try to load them again.
The "extra" stuff is loaded by AppLoader and that's below the ScalaProvider.
ScalaProvider hardcodes parentLoader to topLoader, which is hardcoded to the JNA provider, and it hardcodes extraClasspath to nothing.
In my last mess, I hacked ScalaInstance to insert a static class loader. That worked and could probably be made to work again, but doesn't work with if non-reloadables are used with/loaded beneath multiple Scala versions.
Not really sure where to go from here. I really don't want to muck with the loader.
Steven Parkes
If there's a better approach, I'm more than happy to lend my efforts to fleshing it out, given a suggested approach.
I don't think I'm in a place to make a proposal at this depth in the overall architecture.
Marc
Steven Parkes
No. It's creating a mechanism to allow loading classes in such a way that they won't get reloaded. While class loading is generally stateless, in some cases, e.g., JDBC drivers and classes that load JNIs, it's not.
Marc
Steven Parkes
The basic idea holds: allow jars to be loaded above all scala-instances. As that's not currently possible, I load them above the single scala instance I care about (since I'm only worried about one at this point).
A related issue is where to put the jars. They can't be put in lib because those jars can get reloaded.
So I've put them in "boot". Which almost leads me to a proposal, that jars in boot get loaded automatically by a classloader somewhere around where the JNA loader is right now.
That actually fits well with my deployment stuff, where I put a single copy of these jars/jnis in a place where a j2 container would only load them once, above all individual apps.
It's close in concept to "provided", but the classloader has to be specialized for this, I think. And since that has to happen to early/so low, I don't have a better idea than hardcoding a directory name. Actually, I think that directory could be customized in project/boot.properities and I wouldn't be against defaulting it to nothing.
Doesn't seem like a totally sucky idea ...
Mark Harrah
I ended up trying out a fix that ideally should work without much configuration. In theory, you put your native libraries on the classpath along with your jars or make it available through java.library.path as usual. sbt uses a custom class loader that copies the library to a temporary location so that the jvm doesn't give the UnsatisfiedLinkError related to multiple class loaders loading the library. Currently, this is used for 'run' and 'test'. If this works for you, we can try to get it into the web plugin as well.
Thanks,
Mark
Steven Parkes
- I guess this is a question: the jvm will allow you load the same shared library multiple times? It doesn't have problem with symbol collisions at the shared library level?
- These are never unloaded, right? If you keep loading the same shared library over and over, aren't you going to eventually crash? I admit not having a clue how long that would take. While reloading is what I would want if I was developing a JNI, mine are static so the reloading isn't providing me any utility but seems to be costing something.
- You mentioned implementing findLibrary. I presume loadLibrary calls this (that's the call javasqlite is making)?
- How do you clean up the copies?
- This doesn't address the jdbc driver issue, which is caused by state in java.sql.DriverManager. I was thinking that could be handled by some unload hooks and java.sql.DriverManager.deregisterDriver but I've been thinking about the new parallelism in 0.10 and I'm not so sure that's going to play well with the statics in java.sql.DriverManager.
Mark Harrah
Steven Parkes <smpa...@smparkes.net> wrote:
> I'm not an expert on the JVM or JNIs, so these comments may not jibe with reality:
>
> - I guess this is a question: the jvm will allow you load the same shared library multiple times? It doesn't have problem with symbol collisions at the shared library level?
The jvm only checks that you haven't loaded a native library from the exact same location.
> - These are never unloaded, right? If you keep loading the same shared library over and over, aren't you going to eventually crash? I admit not having a clue how long that would take. While reloading is what I would want if I was developing a JNI, mine are static so the reloading isn't providing me any utility but seems to be costing something.
Native libraries are unloaded when their class loader is garbage collected, so this shouldn't be a problem.
> - You mentioned implementing findLibrary. I presume loadLibrary calls this (that's the call javasqlite is making)?
Yes.
http://java.sun.com/docs/books/jni/html/design.html#6814
> - How do you clean up the copies?
They are deleted after task execution.
> - This doesn't address the jdbc driver issue, which is caused by state in java.sql.DriverManager. I was thinking that could be handled by some unload hooks and java.sql.DriverManager.deregisterDriver but I've been thinking about the new parallelism in 0.10 and I'm not so sure that's going to play well with the statics in java.sql.DriverManager.
I don't know if I missed this part or forgot it, but no, this isn't addressed.
-Mark
Steven Parkes
Thanks for the help.