EC2 slave connectivity issues

736 views

Skip to first unread message

Toni Van de Voorde

unread,

Aug 29, 2019, 7:05:20 AM8/29/19

to Jenkins Users

Hey everyone,

I suddenly have issues with my master slave setup on jenkins. It seems there is a connectivity issue each time a job wants to run on it. I already opened a ticket on Jira for ec2-plugin (https://issues.jenkins-ci.org/browse/JENKINS-59137) but not knowing if the problem is really in that plugin, or if they uberhaubt are going to answer quickly, I'm trying my luck here, in the hope someone already encountered this.

As previously mentioned, AFAIK the problem suddenly happened ... I did upgrade to latest versions when the problem started, but the it didn't solve it.

So what's really happening is that whenever a job needs to a run, the master starts a slave and executes the job ... but the job fails in 2 use cases:

1. The job fails immediately

Branch event
Obtained Jenkinsfile from 140cfe60aa7963ef13cb23c18f07bf70ee97d061
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] stage
[Pipeline] { (php 5.6 and php 7)
[Pipeline] parallel
[Pipeline] { (Branch: php 5.6)
[Pipeline] { (Branch: php 7)
[Pipeline] stage
[Pipeline] { (php 5.6)
[Pipeline] stage
[Pipeline] { (php 7)
[Pipeline] node
[Pipeline] node
Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) in /home/ec2-user/workspace/adsdaq_portal_master
[Pipeline] {
[Pipeline] checkout
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
Running on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) in /home/ec2-user/workspace/adsdaq_portal_master
[Pipeline] {
[Pipeline] }
Failed in branch php 5.6
[Pipeline] checkout
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
Failed in branch php 7
[Pipeline] // parallel
[Pipeline] }

This job above runs in parallel and even executes 2 slaves. It happens that both fail (like here), but sometimes one passes.

The additional stack trace we get in this use case is:

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx)
                at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
                at hudson.remoting.Channel.call(Channel.java:955)
                at hudson.FilePath.act(FilePath.java:1072)
                at hudson.FilePath.act(FilePath.java:1061)
                at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
                at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
                at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
                at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
                at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
                at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
                at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
                at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8
        at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
        at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167)
        at hudson.remoting.Channel.export(Channel.java:768)
        at hudson.remoting.Channel.export(Channel.java:731)
        at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204)
        at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
        at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
        at hudson.remoting.UserRequest.perform(UserRequest.java:222)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
Also:           Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx)
                        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
                        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
                        at hudson.remoting.Channel.call(Channel.java:955)
                        at hudson.FilePath.act(FilePath.java:1072)
                        at hudson.FilePath.act(FilePath.java:1061)
                        at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
                        at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
                        at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
                        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
                        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
                        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
                        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
                        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.lang.NoClassDefFoundError: Could not initialize class com.sun.proxy.$Proxy8
                at sun.reflect.GeneratedConstructorAccessor5.newInstance(Unknown Source)
                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
                at java.lang.reflect.Proxy.newProxyInstance(Proxy.java:739)
                at hudson.remoting.RemoteInvocationHandler.wrap(RemoteInvocationHandler.java:167)
                at hudson.remoting.Channel.export(Channel.java:768)
                at hudson.remoting.Channel.export(Channel.java:731)
                at org.jenkinsci.plugins.gitclient.LegacyCompatibleGitAPIImpl.writeReplace(LegacyCompatibleGitAPIImpl.java:204)
                at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at java.io.ObjectStreamClass.invokeWriteReplace(ObjectStreamClass.java:1230)
                at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1136)
                at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
                at hudson.remoting.UserRequest._serialize(UserRequest.java:263)
                at hudson.remoting.UserRequest.serialize(UserRequest.java:272)
                at hudson.remoting.UserRequest.perform(UserRequest.java:222)
                at hudson.remoting.UserRequest.perform(UserRequest.java:54)
                at hudson.remoting.Request$2.run(Request.java:369)
                at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) failed
Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxxx) failed
        at hudson.remoting.Channel.call(Channel.java:961)
        at hudson.FilePath.act(FilePath.java:1072)
        at hudson.FilePath.act(FilePath.java:1061)
        at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
        at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
        at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
        at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
        at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
        at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

2. The job executes but fails when recording the junits

Another scenario which happens (randomly) is that the execution starts correctly, but when all steps were performed, the job fails from the moment the junits need to be recorded:

 [30;42mOK (715 tests, 2105 assertions) [0m

Generating code coverage report in Clover XML format ... done

Generating code coverage report in HTML format ... done

Remaining indirect deprecation notices (17)

Other deprecation notices (7)
[Pipeline] step
Publishing Clover coverage report...
Publishing Clover HTML report...
Publishing Clover XML report...
Processing Clover XML report ...
Publishing Clover coverage results...
[Pipeline] junit
Recording test results
Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-0617d5ac254d28fce) failed

The stack trace provide after is:

java.io.FileNotFoundException: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-0.amzn2.x86_64/jre/lib/resources.jar
	at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:938)
	at sun.misc.URLClassPath$JarLoader.access$800(URLClassPath.java:801)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:886)
	at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:879)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:878)
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxx)
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:955)
		at hudson.FilePath.act(FilePath.java:1072)
		at hudson.FilePath.act(FilePath.java:1061)
		at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
		at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
		at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Caused: java.lang.InternalError
	at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1052)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:249)
	at sun.misc.URLClassPath.getResource(URLClassPath.java:302)
	at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1264)
	at java.lang.ClassLoader.getResource(ClassLoader.java:1093)
	at java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:233)
	at java.util.ResourceBundle$RBClassLoader.getResourceAsStream(ResourceBundle.java:524)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2686)
	at java.util.ResourceBundle$Control$1.run(ResourceBundle.java:2671)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.util.ResourceBundle$Control.newBundle(ResourceBundle.java:2670)
	at java.util.ResourceBundle.loadBundle(ResourceBundle.java:1510)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1474)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428)
	at java.util.ResourceBundle.findBundle(ResourceBundle.java:1428)
	at java.util.ResourceBundle.getBundleImpl(ResourceBundle.java:1370)
	at java.util.ResourceBundle.getBundle(ResourceBundle.java:854)
	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:169)
	at com.sun.org.apache.xerces.internal.utils.SecuritySupport$7.run(SecuritySupport.java:166)
	at java.security.AccessController.doPrivileged(Native Method)
	at com.sun.org.apache.xerces.internal.utils.SecuritySupport.getResourceBundle(SecuritySupport.java:166)
	at com.sun.org.apache.xerces.internal.util.SAXMessageFormatter.formatMessage(SAXMessageFormatter.java:58)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.getProperty(AbstractSAXParser.java:2063)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.setProperty(SAXParserImpl.java:574)
	at org.dom4j.io.SAXHelper.setParserProperty(SAXHelper.java:34)
	at org.dom4j.io.SAXReader.configureReader(SAXReader.java:911)
	at org.dom4j.io.SAXReader.read(SAXReader.java:463)
	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:348)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:206)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:178)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:143)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
Caused: java.io.IOException: Remote call on EC2 (jenkins-slave) - Adsdaq Jenkins Slave (i-xxx) failed
	at hudson.remoting.Channel.call(Channel.java:961)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

As you can see these are very strange errors. I repeat that we didn't change anything in the structure of the configuration. Not sure if it's a problem with ec2-plugin, or a change somewhere in amazon.

If anybody has or had this issue and knows how to solve it, I'm all ears, as this is blocking our CD flow :/

Thanks in advance

Cheers

Toni

Toni Van de Voorde

unread,

Sep 2, 2019, 9:04:50 AM9/2/19

to Jenkins Users

Hello all,

I was able to solve my problem. If you are interested find the full explanation on the ticket itself: https://issues.jenkins-ci.org/browse/JENKINS-59137?focusedCommentId=374533&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-374533