Class loading questions

[T.J. Crowder]

Hi Tim,

Welcome back from "holiday" (I know moving house is no vacation).

You said to repost if there were questions raised while you were gone that were still outstanding. I have some questions about class loading which are dotted throughout the "Loading JDBC Drivers from JavaScript" thread (https://groups.google.com/d/topic/vertx/_oJQaeH07Sg/discussion).

The questions boil down to:

1. Why is it that if I create a jar with an org.example.foo.Nifty in it, and put that jar in the lib directory of my module alongside the MySQL Connector/J jar, this JavaScript code in the module main:

try {
    stdout.println("typeof org.example.foo.Nifty: " + typeof org.example.foo.Nifty);
    stdout.println("Loading org.example.foo.Nifty");
    java.lang.Class.forName("org.example.foo.Nifty");
    stdout.println("typeof com.mysql.jdbc.Driver: " + typeof com.mysql.jdbc.Driver);
    stdout.println("Loading com.mysql.jdbc.Driver");
    java.lang.Class.forName("com.mysql.jdbc.Driver");
}
catch (e) {
    stdout.println("Exception: " + (e.message || e.toString()));
}

...produces this output:

typeof org.example.foo.Nifty: function

Loading org.example.foo.Nifty

typeof com.mysql.jdbc.Driver: function

Loading com.mysql.jdbc.Driver

Exception: java.lang.ClassNotFoundException: com.mysql.jdbc.Driver

? Why would it find the class in one jar but not the other? Why can it find it when making the Java packages available to JavaScript, but not when doing Class.forName?

2. If I explicitly create a driver instance in the JavaScript main:

var driver = new com.mysql.jdbc.Driver();

...then in theory, the static init of the Driver class should have occurred. That static init from the MySQL Connector/J source in com.mysql.jdbc.Driver.java looks like this:

static {
    try {
        java.sql.DriverManager.registerDriver(new Driver());
    } catch (SQLException E) {
        throw new RuntimeException("Can't register driver!");
    }
}

...and so I'd expect that this following line of JavaScript code in my main:

conn = java.sql.DriverManager.getConnection(
    "jdbc:mysql://localhost/jdbcexample",
    "username",
    "password"
);

...would be referring to the same DriverManager that the static init in the Driver class registered with. But it would seem not, because it fails saying it can't find a registered driver for the URL. Why is that?

3. Why is it that if I create another jar with this compiled class in it:

package com.farsightsoftware;

import java.sql.Connection;
import java.sql.SQLException;
import java.sql.DriverManager;

public class DriverBridge {

    public static final void loadDriver(String cls)
    throws ClassNotFoundException {
        Class.forName(cls);
    }

    public static final Connection getConnection(String url, String user, String pass)
    throws SQLException {
        return DriverManager.getConnection(url, user, pass);
    }
}

...then this JavaScript code in main *works*

stdout.println("Loading driver");
com.farsightsoftware.DriverBridge.loadDriver("com.mysql.jdbc.Driver");
stdout.println("Getting connection");
conn = com.farsightsoftware.DriverBridge.getConnection(
    "jdbc:mysql://localhost/jdbcexample",
    "username",
    "password"
);

? The driver gets found by the Java call to Class.forName, and the static init registers with the DriverManager that the Java code sees, and all is well.

Just to be clear, none of this is meant as disguised criticism (questions like this can seem that way, particularly from someone you don't know). I'm just trying to understand the environment. It could be that one or more of these issues points up something worth changing in the platform, but equally it could be that once you understand it, it makes sense.

I suspect all of the above stem from something fundamental, possibly even something simple, that I'm just not getting. Probably two distinct things, actually: One for why one class is found and the other isn't (question #1), and another related to the other questions. I suspect part of it relates to different classloaders being used for making the Java classes available to JavaScript and loading classes from jars in the lib directory, but that's speculation, and even two different classloaders doesn't quite seem (to me) to explain all of the above.

Thanks in advance,

-- T.J.

[Tim Fox]

As I wild guess, perhaps the JDBC driver is using the thread context classloader. There is a long standing bug in vert.x whereby the tccl is not set correctly, so anything that relies on that might fail.

I actually fixed that bug earlier this week, but won't commit until next week due to my lack of network.

But without debugging this it's hard to tell for sure.

--
You received this message because you are subscribed to the Google Groups "vert.x" group.
To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/p_2GO3VpDwgJ.
To post to this group, send an email to ve...@googlegroups.com.
To unsubscribe from this group, send email to vertx+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/vertx?hl=en-GB.

[Brian Lalor]

On Sep 19, 2012, at 3:51 AM, Tim Fox <timv...@gmail.com> wrote:

On 18/09/2012 13:25, T.J. Crowder wrote:

try {
    stdout.println("typeof org.example.foo.Nifty: " + typeof org.example.foo.Nifty);
    stdout.println("Loading org.example.foo.Nifty");
    java.lang.Class.forName("org.example.foo.Nifty");
    stdout.println("typeof com.mysql.jdbc.Driver: " + typeof com.mysql.jdbc.Driver);
    stdout.println("Loading com.mysql.jdbc.Driver");
    java.lang.Class.forName("com.mysql.jdbc.Driver");
}


As I wild guess, perhaps the JDBC driver is using the thread context classloader. There is a long standing bug in vert.x whereby the tccl is not set correctly, so anything that relies on that might fail.

He's directly loading the class with Class.forName() however.

[T.J. Crowder]

Hi,

On Wednesday, 19 September 2012 08:51:57 UTC+1, Tim Fox wrote:

As I wild guess, perhaps the JDBC driver is using the thread context classloader. There is a long standing bug in vert.x whereby the tccl is not set correctly, so anything that relies on that might fail.

I actually fixed that bug earlier this week, but won't commit until next week due to my lack of network.

But without debugging this it's hard to tell for sure.

Thanks.

First, briefly: Please completely disregard question #1 (why can java.lang.Class.forName find "org.example.foo.Nifty" but not "com.mysql.jdbc.Driver"). Experimental error, the Nifty class wasn't being loaded from the jar at all. I had the class file lying around relative to the current directory (doh!). Removing the class file makes java.lang.Class.forName fail to load either Nifty or Driver. That's a relief, really didn't like it loading one and not the other.

Re questions 2 and 3:

I've been able to dig into this a bit more. If your tccl fix makes the JavaScript call to java.lang.Class.forName work, then it will probably fix the problem. I've attached an unzip-and-run test case so we can easily find out. If it doesn't, I may have the answer.

The problem is that DriverManager reaches into the call stack and does something bordering on evil: Even though DriverManager.registerDriver accepts a driver instance, before it will use that instance for anything (whether it's getConnection, getDriver, getDrivers [an enumeration], etc.), it first checks that the "caller's classloader" will load the driver class. It does this by using a native method to get the "caller's classloader" and then calling this function:

private static boolean isDriverAllowed(Driver driver, ClassLoader classLoader) {
    boolean result = false;
    if(driver != null) {
        Class<?> aClass = null;
        try {
            aClass =  Class.forName(driver.getClass().getName(), true, classLoader);
        } catch (Exception ex) {
            result = false;
        }

         result = ( aClass == driver.getClass() ) ? true : false;
    }

    return result;
}

So even a registered driver instance will be put aside if the caller's classloader doesn't load the class. So the question becomes: What do they mean by "caller's classloader?" One of the comments is "Gets the classloader of the code that called this method" from which I suspect they mean the classloader that loaded the class containing the method that made the call to DriverManager. I didn't know what classloader that is in the case of our JavaScript code that's been compiled to a generated bytecode class -- the classloader that loaded the Rhino jar? The one that loaded RhinoVerticle? It's clearly not the classloader that loads my DriverBridge class (although the attached test case seems to suggest it is, oddly), because DriverManager works correctly when I call it via my DriverBridge class. Searching around to find out what classloader the JavaScript code uses lead me to Rhino's ContextFactory#getApplicationClassLoader[1] And that tells me that unless someone has changed it via initApplicationClassLoader, Rhino will use...the current thread's context classloader.

So here's hoping. If the tccl fix doesn't do it, it may be that initApplicationClassLoader is the key.

-- T.J.

[1] http://www-archive.mozilla.org/rhino/apidocs/org/mozilla/javascript/ContextFactory.html#getApplicationClassLoader() 

[Pid *]

On 19 Sep 2012, at 18:27, "T.J. Crowder" <t...@crowdersoftware.com> wrote:

Hi,

On Wednesday, 19 September 2012 08:51:57 UTC+1, Tim Fox wrote:

As I wild guess, perhaps the JDBC driver is using the thread context classloader. There is a long standing bug in vert.x whereby the tccl is not set correctly, so anything that relies on that might fail.

I actually fixed that bug earlier this week, but won't commit until next week due to my lack of network.

But without debugging this it's hard to tell for sure.

Thanks.

First, briefly: Please completely disregard question #1 (why can java.lang.Class.forName find "org.example.foo.Nifty" but not "com.mysql.jdbc.Driver"). Experimental error, the Nifty class wasn't being loaded from the jar at all. I had the class file lying around relative to the current directory (doh!). Removing the class file makes java.lang.Class.forName fail to load either Nifty or Driver. That's a relief, really didn't like it loading one and not the other.

Re questions 2 and 3:

I've been able to dig into this a bit more. If your tccl fix makes the JavaScript call to java.lang.Class.forName work, then it will probably fix the problem. I've attached an unzip-and-run test case so we can easily find out. If it doesn't, I may have the answer.

The problem is that DriverManager reaches into the call stack and does something bordering on evil:

There is a known inherent design fault in DriverManager, when considered from the position of an application container with an inverted classloader hierarchy.

The Tomcat project evolved memory leak detection/protection to address this, amongst other JRE issues. From vert.x point of view, when this works, the driver must be unloaded when the verticle stops, to avoid creating a classloader memory leak.

p

Even though DriverManager.registerDriver accepts a driver instance, before it will use that instance for anything (whether it's getConnection, getDriver, getDrivers [an enumeration], etc.), it first checks that the "caller's classloader" will load the driver class. It does this by using a native method to get the "caller's classloader" and then calling this function:

private static boolean isDriverAllowed(Driver driver, ClassLoader classLoader) {
    boolean result = false;
    if(driver != null) {
        Class<?> aClass = null;
        try {
            aClass =  Class.forName(driver.getClass().getName(), true, classLoader);
        } catch (Exception ex) {
            result = false;
        }

         result = ( aClass == driver.getClass() ) ? true : false;
    }

    return result;
}

So even a registered driver instance will be put aside if the caller's classloader doesn't load the class. So the question becomes: What do they mean by "caller's classloader?" One of the comments is "Gets the classloader of the code that called this method" from which I suspect they mean the classloader that loaded the class containing the method that made the call to DriverManager. I didn't know what classloader that is in the case of our JavaScript code that's been compiled to a generated bytecode class -- the classloader that loaded the Rhino jar? The one that loaded RhinoVerticle? It's clearly not the classloader that loads my DriverBridge class (although the attached test case seems to suggest it is, oddly), because DriverManager works correctly when I call it via my DriverBridge class. Searching around to find out what classloader the JavaScript code uses lead me to Rhino's ContextFactory#getApplicationClassLoader[1] And that tells me that unless someone has changed it via initApplicationClassLoader, Rhino will use...the current thread's context classloader.

So here's hoping. If the tccl fix doesn't do it, it may be that initApplicationClassLoader is the key.

-- T.J.

[1] http://www-archive.mozilla.org/rhino/apidocs/org/mozilla/javascript/ContextFactory.html#getApplicationClassLoader() 

--

You received this message because you are subscribed to the Google Groups "vert.x" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/pxaBnv9n_BoJ.

[chiquitinxx]

Hey!

I have same error. Driver not found. Running vertx with -cp "xxxjdbcdriver.jar" dont works. But if you copy that jar to $VERTX_HOME/lib then, it works! So its a problem with the classloader, for me in Groovy. But i can't upload jar to vertx in heroku. Need that fix :D

Thanks!

[T.J. Crowder]

Hi,

I suspect your problem is different from that outlined in my scenario, which appears to be specific to JavaScript support. If you wrap your Groovy code into a module (e.g., put the compiled class file in  mods/your-mod-name, create a mods/your-mod-name/mod.json, and put the driver jar in a mods/your-mod-name/lib directory), I expect it'll get found. That's my experience with compiled Java code, anyway, and of course once compiled it's just...bytecode.

But I know nothing about Groovy. :-)

-- T.J.

[Pid *]

On 24 Sep 2012, at 01:18, chiquitinxx <jorge.fr...@gmail.com> wrote:

Hey!

I have same error. Driver not found. Running vertx with -cp "xxxjdbcdriver.jar" dont works. But if you copy that jar to $VERTX_HOME/lib then, it works! So its a problem with the classloader, for me in Groovy. But i can't upload jar to vertx in heroku. Need that fix :D

How do you upload your verticle scripts?

Can you upload a module and put it in the module lib?

p

Thanks!

--
You received this message because you are subscribed to the Google Groups "vert.x" group.

To view this discussion on the web, visit https://groups.google.com/d/msg/vertx/-/bHkjMZm0dxcJ.