Hi, All -
Maybe this is answered in previous messages or in the
documentation, but I can't find it. Does NestedVM create Java classes
that represent shared libraries or does it create classes that
represent statically linked applications?
My reason for asking is that I have a number of legacy C-language
libraries that I would like to access from within a Java-based
application server (Spring Framework/ServiceMix/Hibernate). While I
can (and have) used SWIG to create Java (and TCL) wrappers for the
libraries, I would have to create a separate library file for each
supported platform (I have at least three - 32-bit Windows
workstations, 32-bit Solaris workstations, and 54-bit Solaris
servers). The perfect solution (I think, possibly based on ignorance)
would be to run each library through the NestedVM toolchain and save
each library's class in a JAR file. Is this something NestedVM can do
today? In principle if someone (i.e., me) puts the work in? Can't do
because ... ?
Thanks for writing a really exciting tool and making it available!
Sorry I have no answers, but I do have additional questions. Do you currently use JNI, JNA, o r something else? When I was seriously looking at NestedVM, it was to replace a bunch of number crunching C code that was being accessed via JNI. So little processor time was spent in the C code that one percent of native speed probably would have been fast enough. Ultimately, I gave up the idea due to lack of NestedVM tool support and documentation.
Considering the huge amount of C, C++, and Fortran code that only does number crunching, well documented platform-independent binaries would be a great thing. It would make platform independent scientific software feasible in many places where it isn't currently.
On Fri, Jan 04, 2008 at 01:59:22PM -0800, David Doty wrote: > documentation, but I can't find it. Does NestedVM create Java classes > that represent shared libraries or does it create classes that > represent statically linked applications?
NestedVM doesn't support shared libraries yet, and there aren't currently any plans for doing so. I don't think there are any compelling reasons for supporting them.
Most users of NestedVM are only using it for a single binary. In these cases there is absolutely no advantage to using shared libraries (in fact, using shared libraries would actually increase the size of the application because the linker couldn't drop dead code).
In the cases where multiple binaries are required the recommended trick is to link them all together as a single binary and dispatch to the correct main() via argv[0]. This gets us all the space advantages of shared libraries at the cost of a little extra complexity.
> My reason for asking is that I have a number of legacy C-language > libraries that I would like to access from within a Java-based > application server (Spring Framework/ServiceMix/Hibernate).
I'd recommend that you statically link all your libraries into a single mips binary and run NestedVM on that. If space usage isn't an issue you could even make one binary per library (at the cost of duplicating libc each time).
Please post any questions you might have on doing this. What you're trying to do is 100% possible (SQLite JDBC does exatcly this) but not documented well. We'd be happy to help.
On Fri, Jan 04, 2008 at 06:06:14PM -0600, Curt Cox wrote: > currently use JNI, JNA, o r something else? When I was seriously > looking at NestedVM, it was to replace a bunch of number crunching C > code that was being accessed via JNI. So little processor time was > spent in the C code that one percent of native speed probably would > have been fast enough. Ultimately, I gave up the idea due to lack of > NestedVM tool support and documentation.
Supporting JNI would definitely be a huge help is easing the transition to NestedVM. This is something I've always hoped someone would get around to implementing someday. The only reason I haven't is because I've never actually used JNI for anything bigger than hello-worldish stuff and I'm not too familiar with it.
However, I think (possibly with a few tweaks and some additional boilerplate code) NestedVM's "native interface" is way better, and easier to use than JNI once you understand it (which probably requires begging for help on the mailing list because it isn't well documented).
Aside from that, what other problems did you run into with NestedVM? We'd be happy to help any way we can.
> On Fri, Jan 04, 2008 at 06:06:14PM -0600, Curt Cox wrote: > > currently use JNI, JNA, o r something else? When I was seriously > > looking at NestedVM, it was to replace a bunch of number crunching C > > code that was being accessed via JNI. So little processor time was > > spent in the C code that one percent of native speed probably would > > have been fast enough. Ultimately, I gave up the idea due to lack of > > NestedVM tool support and documentation.
> Supporting JNI would definitely be a huge help is easing the transition > to NestedVM. This is something I've always hoped someone would get > around to implementing someday. The only reason I haven't is because > I've never actually used JNI for anything bigger than hello-worldish > stuff and I'm not too familiar with it.
I've also given some thought to NestedVM JNI support.
How do you envision NestedVM integrating with JNI?
Build a custom NestedVM JNI library to interface with MIPS object code transparently in a pure 100% java solution? Wouldn't NestedVM have to support ELF MIPS shared libraries and dlopen/dlsym for this to work? Or just fake the loading of shared MIPS libraries via some statically compiled NestedVM/MIPS convention?
> However, I think (possibly with a few tweaks and some additional > boilerplate code) NestedVM's "native interface" is way better, and > easier to use than JNI once you understand it (which probably requires > begging for help on the mailing list because it isn't well documented).
___________________________________________________________________________ _________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs
On Mon, Jan 07, 2008 at 10:41:15AM -0800, Joe Wilson wrote: > How do you envision NestedVM integrating with JNI?
I wouldn't really call it integrating with JNI. More like reimplementing JNI on top of the (very low level) _call_java()/Runtime.call() interface. This would have nothing to do with the "real" JNI, it would just have the same API.
Supporting call into Java from C should be pretty straightforward. On the java side you'd use normal reflection stuff to find classes, methods, and objects. They would have to be stored in an array so that we could return an int "handle" back to NestedVM. You'd have a CallJavaCB object (the Java code that sits behind _call_java()) that implements all this. On the C side you'd have to reimplement all the JNI functions in terms of this new interface.
Supporting calls into C from Java ("native" methods) wouldn't be 100% transparent though. "public native int foo(int bar);" would probably have to be rewritten as:
public int foo(int bar) { return JNI.call("MyClass_bar",this,bar); }
where JNI.call magics up a JNIEnv and a handle for this, then does a Runtime.call(env,this_handle,bar);
If we wanted to get really fancy we could use a custom classloader to rewrite "native" methods in this form.
> transparently in a pure 100% java solution? Wouldn't NestedVM have to > support ELF MIPS shared libraries and dlopen/dlsym for this to work?
The "libraries" would just be plain old statically linked binaries with a dummy main() and a ton of dead code (the actual library). You'd never run them as a standalone application. You only access them via Runtime.call().
Shared libraries appear to be a more elegant way of handling this but supporting them means we have to deal with position independent code, runtime relocations, and everything else done by the dynamic linker in a real system. While this is all certainly possible it doesn't seem worth the effort.
> On Mon, Jan 07, 2008 at 10:41:15AM -0800, Joe Wilson wrote: > > How do you envision NestedVM integrating with JNI? > Supporting calls into C from Java ("native" methods) wouldn't be 100% > transparent though. "public native int foo(int bar);" would probably > have to be rewritten as:
> public int foo(int bar) { return JNI.call("MyClass_bar",this,bar); }
> where JNI.call magics up a JNIEnv and a handle for this, then does a > Runtime.call(env,this_handle,bar);
> If we wanted to get really fancy we could use a custom classloader to > rewrite "native" methods in this form.
To avoid rewriting working JNI java source files, this step would be needed. But you don't have to do this method rewriting at runtime with a classloader. It could be done at compile time - after javac is run on the java source files containing the native methods. A standalone java program could be written to perform this task.
Does classgen support loading existing class files, rewriting individual methods and dumping the entire class file out again?
On Tue, Jan 08, 2008 at 12:09:39AM -0800, Joe Wilson wrote: > Does classgen support loading existing class files, rewriting > individual methods and dumping the entire class file out again?
Yep. We've used it for that in a few different projects.
Hi, Brian -
Thanks for your quick reply. It is really valuable to hear what
the intended use is for a tool. To amplify some on my initial
message, here are the two things I'd like to be able to do :
o Call C-language TCL extensions from JACL (the pure-Java
version of TCL). There are a huge number of these, and
most won't be translated to Java any time soon (if ever).
o Convert a legacy C application into a shared library and
call its functions from C, TCL, or Java applications.
The first of these appears to be possible with JNA (as Curt
mentioned), so I won't address it further. The second sounds a lot
like the SQLlite case.
I provide programming support for folks who work with satellite
orbits. We get satellite orbits as "two-line element sets" (TLEs).
We have an old C program that reads a TLE from standard input and
writes the "crossings" (i.e., where and when the satellite will cross
the equator) to standard output. It does this for exactly one TLE per
execution (after all, when the C code was originally written, you ran
it from the command line - how many TLEs could you type?).
We have some Java code that executes this app to calculate
crossings on demand. The usual scenario requires calculating dozens
(or, occasionally, hundreds) of crossings. At the moment, each of
these requires a separate execution of the C program, with the
attendant time taken for dozens of load-the-process-execute-the-
process-unload-the-process cycles. People grow old waiting ... ;-)
(We also have users who would like to get crossings from TCL or Python
scripts, other C/C++/Java language applications, etc. but that may be
a different issue).
While I could modify the C app to process an arbitrary number of
crossings per execution, this would make it harder to use. I suppose
I could wrap it in something and make a web service out of it, but
that seems extreme. My first thought was to turn the guts of the C
app into a shared library and then use it (either directly or via a
SWIG wrapper) for each of the programming languages that want access.
This leads to the build-it-on-all-target-platforms problem, which I'd
really rather avoid. Since there are Java-based implementations of
many major scripting languages (e.g., JACL, JRuby, Jython), I figured
that a reasonably step would be to provide the library in a form that
can be called from any Java-based platform. Hence my interest in
NestedVM.
It sounds like I should take a look at how SQLlite does its JDBC as
an example another method to make the crossings library accessible on
the Java side, right?
> Sorry I have no answers, but I do have additional questions. Do you
> currently use JNI, JNA, or something else?
My thought was to use SWIG to wrap the C code for various platforms
(esp. TCL and Java). For Java, SWIG generates the JNI glue code as
well as the Java class to wrap it. I'd like to avoid having to
compile separate versions of the library for each platform, which
makes the NestedVM concept really attractive. Since there is a JVM-
based version of TCL (as well as several other important scripting
languages), JNA sounds like a real possibility - thanks for pointing
it out (I'd never heard of it).
> Considering the huge amount of C, C++, and Fortran code that only does
> number crunching, well documented platform-independent binaries would
> be a great thing. It would make platform independent scientific
> software feasible in many places where it isn't currently.
Absolutely! Who wants to re-write litterally billions of lines of
code??
Hi, Brian -
I spent a little time looking at JNA per Curt's post. It looks
like a really promising technology. Basically, instead of writing a
bunch of JNI code to a C (or C++) library, you describe the interfaces
exposed by that library and the JNA runtime generates the equivalent
of the JNI, allowing you to call the library. It appears to support
calling back from the C side to the Java side, which you mention
elsewhere as a big issue. The hard part is hand-crafting the Java
code to describe the C interfaces...
Fortunately, *that* problem has been solved by the SWIG folks.
SWIG can read the C header files and create a wrapper for a couple of
dozen different scripting languages and other run-times (e.g., JNI).
It can also produce an XML description of the interfaces. It seems to
me that it wouldn't be difficult to transform the XML into the
corresponding JNA API calls and - la voila - instant Java interface to
any library with a C-language API.
But ... we still have the problem of compiling, linking, and
distributing all the platform-specific variations of the library.
*This* is where NestedVM could complete the picture - if NestedVM
could support JNA as the generic alternative to finding the platform-
specific version of the library. We could now distribute any library
that
o Is compilable by the GCC suite and
o Exposes a C-language API
in a form executable on any platform that supports Java. And never
have to compile a platform-specific version again.
This would, I think, require that NestedVM to support treating each
shared library separately, rather than packaging all of them together
in a single executable, since in the general case we would want to use
arbitrary combinations of libraries.
Does this seem feasible, or have I been smoking too much weed?
> On Tue, Jan 08, 2008 at 12:09:39AM -0800, Joe Wilson wrote: > > Does classgen support loading existing class files, rewriting > > individual methods and dumping the entire class file out again?
> Yep. We've used it for that in a few different projects.
> -Brian
In the classgen ClassFile constructor I see this note:
// FEATURE: Support these // NOTE: Until we can support them properly we HAVE to delete them, // they'll be incorrect after we rewrite the constant pool, etc attrs.remove("InnerClasses");
Can classgen handle reading and rewriting inner classes correctly?
Also, do you see any issue with using classgen on bytecode with java generics?
