Classloading issues between git and git-client plugins?!

[Sandell, Robert]

Onto my next post LTS upgrade problem.

 

Suddenly we see this stacktrace in build logs when the git plugin is fetching from the repository.

 

Caused by: java.lang.NoClassDefFoundError: org.jenkinsci.plugins.gitclient.GitClient

         at java.lang.Class.getDeclaredFields0(Native Method)

         at java.lang.Class.privateGetDeclaredFields(Class.java:2397)

         at java.lang.Class.getDeclaredField(Class.java:1946)

         at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1659)

         at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)

         at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:480)

         at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468)

         at java.security.AccessController.doPrivileged(Native Method)

         at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468)

         at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365)

         at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)

         at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)

         at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)

         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)

         at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

         at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

         at hudson.remoting.UserRequest.deserialize(UserRequest.java:182)

         at hudson.remoting.UserResponse.retrieve(UserRequest.java:211)

         at hudson.remoting.Channel.call(Channel.java:723)

 

A similar stacktrace if we try to run the build on the master instead.

 

We got this problem at one time when we first upgraded (everything), my first guess was that it was because of some library changes in git-client so we cleaned up some old hpi and jpi files and installed the previous micro version of git and git client, started Jenkins and it worked again.

The other servers didn’t have the problem when we upgraded them. But suddenly yesterday it happened in runtime. On Monday the git plugin worked and on Tuesday it didn’t.

 

I’ve tried some Groovy debugging with the following script:

 

def gitClient = Jenkins.instance.pluginManager.getPlugin("git-client")

println(gitClient.classLoader.findLoadedClass("org.jenkinsci.plugins.gitclient.GitClient"))

//output: interface org.jenkinsci.plugins.gitclient.GitClient 

println(Jenkins.instance.pluginManager.uberClassLoader.findClass("org.jenkinsci.plugins.gitclient.GitClient"))

//output: interface org.jenkinsci.plugins.gitclient.GitClient

 

def gitPlugin = Jenkins.instance.pluginManager.getPlugin("git") 

println(gitPlugin.classLoader.findLoadedClass("hudson.plugins.git.GitSCM"))

//output: class hudson.plugins.git.GitSCM

println(Jenkins.instance.pluginManager.uberClassLoader.findClass("hudson.plugins.git.GitSCM")) 

//output: class hudson.plugins.git.GitSCM  

println(gitPlugin.classLoader.findLoadedClass("org.jenkinsci.plugins.gitclient.GitClient"))

//output: interface org.jenkinsci.plugins.gitclient.GitClient

 

def job = Jenkins.instance.getItem("Tools_GerritTester") 

println(job.getClass().getClassLoader().findLoadedClass("org.jenkinsci.plugins.gitclient.GitClient")) 

//output: null  

println(job.getClass().getClassLoader().findLoadedClass("hudson.plugins.git.GitSCM"))   

//output: null  

println(job.getScm().getClass().getClassLoader().findLoadedClass("org.jenkinsci.plugins.gitclient.GitClient"))

//output: interface org.jenkinsci.plugins.gitclient.GitClient  

 

 

The output is the same whether I run it on a master with the problem or one that doesn’t have it.

 

Are there any other classloading scenarios that I’ve missed, or something else I can try to find the issue? I’d prefer to not just reboot and forget about it ;)

 

 

 

Robert Sandell

Staff Engineer

Development Environment

Software Environment and Product Configuration

 

Sony Mobile Communications

Tel: +46 10 80 12721

robert....@sonymobile.com

sonymobile.com

 

 

 

[Mark Waite]

It's the credentials plugin installed, and current, and enabled?

There have been other reports of a class loader failure, but we have not been able to duplicate them.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Waite]

In case it helps, https://issues.jenkins-ci.org/browse/JENKINS-21520 reports something similar, though the investigation there has not shown any way to repeat the bug.

Likewise https://issues.jenkins-ci.org/browse/JENKINS-20254 seems to be in a similar area.

--

Thanks!

Mark Waite

[Sandell, Robert]

Yes those look like similar issues.

Any thoughts on how I can continue to debug the issue?

Few jobs on the currently affected masters use the git plugin, so I can keep them alive for a little bit more before I need to take drastic measures.

 

 

 

Robert Sandell

Sony Mobile Communications

Tel: +46 10 80 12721

sonymobile.com

[Mark Waite]

Unfortunately, I don't have any reasonable suggestions for further techniques to debug the classloading issue.  I am hopeful that Kohsuke or Jesse or Stephen have suggestions, since I think classloading is handled deep in core, well below the layers of my comfort.

As far as I know, most of the jar file dependencies are "up to date" in the git-client-plugin pom.xml and the git-plugin pom.xml.

There are two dependencies intentionally pointed at older versions.  The "guava" dependency is kept at 11.0.01 because Jenkins core keeps its version at 11.0.1.  The multiple-scms dependency is kept at 0.2 because there is a known problem with the multiple-scms plugin 0.3.  Do you have the multiple SCMs plugin installed?  Is it enabled?  Are you using it?

Mark Waite

Mark Waite

[Stephen Connolly]

I have nothing jumping to mind... this looks like something to do with remoting though... in other words the class is failing to load on the remote classloader not Jenkins core's classloader itself.... which probably points to Kohsuke as the best placed to comment. Robert, you could always open a support ticket if you have a support contract with somebody... ;-)

[Sandell, Robert]

The only support contract I have is myself and my groovy script console J

 

What baffles me is that the uberClassLoader can load the class, but the build fails to.

 

 

 

Robert Sandell

Sony Mobile Communications

Tel: +46 10 80 12721

sonymobile.com

 

[Stephen Connolly]

because the channel is not hooked up to the überclassloader somehow... so when the class is called for on the remote end, it doesn't show up. I suspect some of the perf improvements in classloading are hitting long running slaves. If my theory is correct disconnecting and reconnecting the slave in question should clear the issue.

[Robert Sandell]

But if that was the issue then we wouldn't see the issue when running on the master I guess, but we do.

[Sandell, Robert]

I’ve got some progress, still no root cause but more hints at least.

It seems to be a problem with static initialization of the GitClient interface. If I in a groovy script just try to print the name of the class;

 

import org.jenkinsci.plugins.gitclient.*

import hudson.plugins.git.*

 

GitClient.class.getName()

 

I get the following interesting stacktrace:

 

java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.gitclient.GitClient

         at java.lang.Class.forName0(Native Method)

         at java.lang.Class.forName(Class.java:190)

         at Script1.class$(Script1.groovy)

         at Script1.$get$$class$org$jenkinsci$plugins$gitclient$GitClient(Script1.groovy)

         at Script1.run(Script1.groovy:15)

         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:580)

         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618)

         at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589)

         at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:150)

         at hudson.util.RemotingDiagnostics$Script.call(RemotingDiagnostics.java:122)

         at hudson.remoting.LocalChannel.call(LocalChannel.java:45)

         at hudson.util.RemotingDiagnostics.executeGroovy(RemotingDiagnostics.java:119)

         at jenkins.model.Jenkins._doScript(Jenkins.java:3427)

         at jenkins.model.Jenkins.doScript(Jenkins.java:3404)

         at sun.reflect.GeneratedMethodAccessor2793.invoke(Unknown Source)

         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

         at java.lang.reflect.Method.invoke(Method.java:606)

         at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)

         …

 

Any other class.getName() that implements GitClient or has it as a member var will just get the standard NoClassDefFoundError probably because of this underlying issue.

A bit worried about the top native code trace though. But the same jvm version is running on the server that works. And why would it need to initialize the class again? Could it have been unloaded somehow? Seems strange if that’s the case.

 

There are some static initialization going on in GitClient, but nothing that seems to not work except the call to GitAPI that fails because GitAPI implements GitClient (circular definition?)

The CredentialsMatcher code in there works if I run it in a groovy script so it can’t be that I guess.

 

CredentialsMatcher CREDENTIALS_MATCHER = CredentialsMatchers.anyOf(

            CredentialsMatchers.instanceOf(StandardUsernamePasswordCredentials.class),

            CredentialsMatchers.instanceOf(SSHUserPrivateKey.class)

            // TODO does anyone use SSL client certificates with GIT?

    );

 

 

Robert Sandell

Sony Mobile Communications

Tel: +46 10 80 12721

sonymobile.com