Upgrading Digital Ocean Cloud Plugin - First run always fails

58 views
Skip to first unread message

Gavin Mogan

unread,
Feb 11, 2021, 2:19:30 AM2/11/21
to Jenkins Developers
I'm trying to fix the plain text credentials issue in the digital ocean plugin. I've updated it to use credentials, on to testing it, and having an agent issue. The droplet spins up, it connects, the job runs, and then errors with a stacktrace (at the end of email). Then if i run it again it runs on the same agent just fine without error.

Its complaining about `java.lang.ClassNotFoundException: javax.servlet.ServletException` does this mean that some do api is returning a servletexception and then remoting can't seralize it? or is it a known error with 2.222.4? am I missing the plugins somewhere in these stacktraces?

Gavin

Started by user unknown or anonymous
Running as SYSTEM
Building remotely on jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff in workspace /jenkins/workspace/testdo
[testdo] $ /bin/sh -xe /tmp/jenkins4548124856065621833.sh
+ whoami
root
+ hostname
jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff
+ uptime
 07:03:51 up 1 min,  0 users,  load average: 2.58, 1.11, 0.42
+ uname -a
Linux jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
FATAL: Remote call on jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff failed
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to channel
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
		at hudson.remoting.Request.call(Request.java:202)
		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
		at com.sun.proxy.$Proxy5.fetch(Unknown Source)
		at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:317)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
		at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
		at hudson.util.ProcessTree.get(ProcessTree.java:432)
		at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
		at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
		at hudson.remoting.UserRequest.perform(UserRequest.java:211)
		at hudson.remoting.UserRequest.perform(UserRequest.java:54)
		at hudson.remoting.Request$2.run(Request.java:369)
		at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
java.lang.ClassNotFoundException: javax.servlet.ServletException
	at hudson.remoting.RemoteClassLoader$ClassLoaderProxy.fetch(RemoteClassLoader.java:811)
	at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:931)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
	at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:21)
	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:998)
		at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1083)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
		at hudson.model.Run.execute(Run.java:1856)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:428)
Caused: java.lang.NoClassDefFoundError: javax/servlet/ServletException
	at hudson.util.ProcessTree.get(ProcessTree.java:432)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
	at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.io.IOException: Remote call on jenkins-do-do-f0a19f9e-351c-48e3-9c45-3ca58d524eff failed
	at hudson.remoting.Channel.call(Channel.java:1004)
	at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1083)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:510)
	at hudson.model.Run.execute(Run.java:1856)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:428)
Finished: FAILURE

Oleg Nenashev

unread,
Feb 11, 2021, 3:52:52 AM2/11/21
to Jenkins Developers
It is a known error fixed in 2.235.

Gavin Mogan

unread,
Feb 11, 2021, 11:12:30 PM2/11/21
to Jenkins Developers
I've tried it with 2.235.5 and 2.263.4 with the same stacktrace.

My PR is up at https://github.com/jenkinsci/digitalocean-plugin/pull/50 I don't see anything I might have changed related to it. I'll try it again without my changes to see if i caused it. I looked at the PR linked inside the jira ticket and it looks like its only core changes.

Gavin

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/ca060873-eeba-4fc4-baa5-6e38aab7f597n%40googlegroups.com.

Gavin Mogan

unread,
Feb 13, 2021, 10:10:37 PM2/13/21
to Jenkins Developers
Okay managed to do a bunch of permutations.

2.60.3 works just fine on both first and second freestyle job
2.263.4 fails on first job, but will succeed on the second if the agent is still running
2.263.4 works fine on first and second with pipeline

UserRequest.getClassLoader(callable) where callable is Launcher$RemoteLauncher$KillTask returns a WebAppClassLoader. Is that possibly why its trying to load ServletError?

I've been trying to attach a debugger and track this down more but Its well outside my java knowledge.
I've tried attaching it to the hpi:run process, and the agent/remoting.jar process. I think the remoting process is the way to do it, but I can't figure out how to get a breakpoint inside of killtask



image.png


Gavin Mogan

unread,
Feb 24, 2021, 4:23:54 PM2/24/21
to Jenkins Developers
The patch is merged, but before I cut a release, I'd like to address the freestyle issue.

Is anyone available to pair with me to diagnose this? Or can I get someone credit to look into it?

Tim Jacomb

unread,
Feb 25, 2021, 3:54:43 AM2/25/21
to Jenkins Developers
I've had a bit of a look at this,

I patched the plugin to enable logging via workDir, unfortunately it didn't give much more than what's in the log for the freestyle build. (https://gist.github.com/timja/4b28f173d5d2925c82781109dce57362)

But it didn't help any more than say enable logging

FTR I've got a very similar error in the slack plugin on freestyle builds when uploading a file from the agent.

Anyone got any tips for debugging this? Specific logs to enable and how to enable them?

The agent is started here:

I patched it to be:
+            String launchString = "java " + jvmOpts + " -jar /tmp/agent.jar -workDir /root/.jenkins/remotingWorkDir";

Thanks
Tim



Gavin Mogan

unread,
Mar 2, 2021, 9:24:46 PM3/2/21
to Jenkins Developers
Okay if anyone else wants to try, you can use my referral link and get yourself a bunch of free credits.https://m.do.co/c/7d6859326b6a

I'm just going to mark it as a known issue in the release notes as I don't have any idea how to debug this any more.

Gavin

Tim Jacomb

unread,
Mar 19, 2021, 1:05:15 PM3/19/21
to Jenkins Developers
Reply all
Reply to author
Forward
0 new messages