So we use Amazon EC2 plugin which created Dynamic slaves.
I will try to describe the issue:
Slave request was made by the server to the EC2 plugin
EC2 plugin brought up a slave and attached for use
A build picked up the slave and failed with the error below
The slave got assigned to another build and that build also failed
The slave doesn't get decommissioned when the below issue happens rather it keeps disrupting all the Jenkins jobs it gets assigned too.
Workaround:
There is a manual workaround to restart the slave but I am looking for an automated way to detect it and either delete the slave or automatically restart the slave.
My point also is that this issue has happened not once but many times with different people so what can be done to fix it?
19:35:30 [WS-CLEANUP] Deleting project workspace...
19:35:30 [WS-CLEANUP] Done
19:35:30 Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to slave (i-123456)
19:35:30 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1737)
19:35:30 at hudson.remoting.UserResponse.retrieve(UserRequest.java:313)
19:35:30 at hudson.remoting.Channel.call(Channel.java:952)
19:35:30 at hudson.FilePath.act(FilePath.java:1009)
19:35:30 at hudson.FilePath.act(FilePath.java:998)
19:35:30 at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
19:35:30 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:821)
19:35:30 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:812)
19:35:30 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1180)
19:35:30 at hudson.scm.SCM.checkout(SCM.java:504)
19:35:30 at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
19:35:30 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
19:35:30 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
19:35:30 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
19:35:30 at hudson.model.Run.execute(Run.java:1727)
19:35:30 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
19:35:30 at hudson.model.ResourceController.execute(ResourceController.java:97)
19:35:30 at hudson.model.Executor.run(Executor.java:429)
19:35:30 java.lang.NoClassDefFoundError: Could not initialize class org.jenkinsci.plugins.gitclient.GitClient
19:35:30 at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.<init>(CliGitAPIImpl.java:253)
19:35:30 at hudson.plugins.git.GitAPI.<init>(GitAPI.java:77)
19:35:30 at org.jenkinsci.plugins.gitclient.Git$1.invoke(Git.java:134)
19:35:30 at org.jenkinsci.plugins.gitclient.Git$1.invoke(Git.java:120)
19:35:30 at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2827)
19:35:30 at hudson.remoting.UserRequest.perform(UserRequest.java:210)
19:35:30 at hudson.remoting.UserRequest.perform(UserRequest.java:53)
19:35:30 at hudson.remoting.Request$2.run(Request.java:364)
19:35:30 at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
19:35:30 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
19:35:30 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
19:35:30 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
19:35:30 at java.lang.Thread.run(Thread.java:748)
19:35:30 Caused: java.io.IOException: Remote call to slave (i-123456) failed
19:35:30 at hudson.remoting.Channel.call(Channel.java:958)
19:35:30 at hudson.FilePath.act(FilePath.java:1009)
19:35:30 Caused: java.io.IOException: remote file operation failed: $HOME/workspace at hudson.remoting.Channel@678f:slave (i-123456)
19:35:30 at hudson.FilePath.act(FilePath.java:1016)
19:35:30 at hudson.FilePath.act(FilePath.java:998)
19:35:30 at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:137)
19:35:30 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:821)
19:35:30 at hudson.plugins.git.GitSCM.createClient(GitSCM.java:812)
19:35:30 at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1180)
19:35:30 at hudson.scm.SCM.checkout(SCM.java:504)
19:35:30 at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
19:35:30 at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
19:35:30 at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
19:35:30 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
19:35:30 at hudson.model.Run.execute(Run.java:1727)
19:35:30 at hudson.matrix.MatrixRun.run(MatrixRun.java:146)
19:35:30 at hudson.model.ResourceController.execute(ResourceController.java:97)
19:35:30 at hudson.model.Executor.run(Executor.java:429)
19:35:30 Retrying after 10 seconds