[JIRA] [core] (JENKINS-31804) Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

[reese0809@gmail.com (JIRA)]

Charese Williams updated an issue

Jenkins / JENKINS-31804 Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang Change By: Charese Williams Summary: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds  never get   hang Add Comment

This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang this is causing me not to be able to sleep at night. I have to go in and manualy delete nodes at all times of the night because the nodes aren't getting terminated or spawned up and the instance cap is being reached and jenkins goes, I already are at my cap, and my builds stack up.

Add Comment

[francisu@gmail.com (JIRA)]

Francis Upton commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

Can you try the EC2 plugin 1.30 snapshot to see if it's any better?

Add Comment

[francisu@gmail.com (JIRA)]

Francis Upton commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

https://jenkins.ci.cloudbees.com/job/plugins/job/ec2-plugin/lastBuild/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.30-SNAPSHOT/ec2-1.30-SNAPSHOT.hpi

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

java.io.IOException: Failed to dynamically deploy this plugin at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1385) at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1163) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Failed to install ec2 plugin at hudson.PluginManager.dynamicLoad(PluginManager.java:474) at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:1381) ... 5 more Caused by: java.io.IOException: Dependency aws-java-sdk (1.10.26) doesn't exist at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:480) at hudson.PluginManager.dynamicLoad(PluginManager.java:464) ... 6 more

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

after restarting jenkins this will cause me to have ot go back to 1.29 until this is fixed SEVERE: Failed Loading plugin ec2

java.io.IOException: Dependency aws-java-sdk (1.10.26) doesn't exist at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:480)

at hudson.PluginManager$2$1$1.run(PluginManager.java:370) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:905) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Add Comment

[francisu@gmail.com (JIRA)]

Francis Upton commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

You have to install the aws-java-sdk plugin.

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams updated an issue

Jenkins / JENKINS-31804

Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

Change By: Charese Williams Attachment: screenshot-1.png

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang can you make it so that the installation of the upgrade will pull in the dependency if it's not already there (user friendly)...see photo, it looks like the aws-sdk does this seamlessly.

Add Comment

[francisu@gmail.com (JIRA)]

Francis Upton commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

This should already be done by Jenkins in the normal course of declaring a dependency on another plugin in the POM. https://wiki.jenkins-ci.org/display/JENKINS/Dependencies+among+plugins I'm not sure why it's not working. Perhaps because the plugin was installed by uploading an hpi file directly and not through the normal means? Might be something to ask on the Jenkins list/IRC channel.

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

well in any rate the only time I can try this out is at off peak hours because this pipeline we created runs 24/7 and developers develop around the clock. I will try this later on this evening with the sdk plugin installed . more to come

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

1. I was able to install it. unfortunately i'm going ot have to roll back. Also the call that is being used to grab every instance id out of my vpc is costly because I have about 600+ running instances that are serving other purposes. still mass failure. See log stack trace added This happened after the demand had went to zero and jenkins was essentially idle for about 2 hours. A developer attempted to push a build which triggered the creation of a node and nothing happened on the gui.

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams updated an issue

Jenkins / JENKINS-31804

Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

stack trace 1.30-SNAPSHOT.hpi installed Change By: Charese Williams Attachment: ec2.txt

Add Comment

[francisu@gmail.com (JIRA)]

Francis Upton commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang Seems to be the same problem as the original. Need to have a core developer consider this. Is the logging too much? I can reduce a lot of this logging to below INFO level. Beyond that, I'm not sure what you mean about the cost of grabbing every instance out of your VPC. It does have to look at all of the instances to determine if it can use an existing instance. But this should be relatively cheap (modulo the tracing).

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

Francis, Yeah the INFO is a lot in the logs. But I wonder why that tempOfflineReason thing is coming back null?

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

still happening A problem occurred while processing the request. Please check our bug tracker to see if a similar problem has already been reported. If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem. If you think this is a new issue, please file a new issue. When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins. The users list might be also useful in understanding what has happened. Stack trace javax.servlet.ServletException: java.lang.RuntimeException: Failed to serialize hudson.model.Node#temporaryOfflineCause for class hudson.slaves.DumbSlave at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:796) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:722) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at com.smartcodeltd.jenkinsci.plugin.assetbundler.filters.LessCSS.doFilter(LessCSS.java:47) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:200) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:178) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:99) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:129) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:123) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:929) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407) at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:200) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: Failed to serialize hudson.model.Node#temporaryOfflineCause for class hudson.slaves.DumbSlave at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82) at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026) at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015) at com.thoughtworks.xstream.XStream.toXML(XStream.java:988) at hudson.XmlFile.write(XmlFile.java:178) at jenkins.model.Nodes.save(Nodes.java:195) at hudson.util.PersistedList.onModified(PersistedList.java:173) at hudson.util.PersistedList.replaceBy(PersistedList.java:85) at hudson.model.Slave.<init>(Slave.java:172) at hudson.plugins.ec2.EC2AbstractSlave.<init>(EC2AbstractSlave.java:128) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:47) at hudson.plugins.ec2.EC2OndemandSlave.<init>(EC2OndemandSlave.java:40) at hudson.plugins.ec2.SlaveTemplate.newOndemandSlave(SlaveTemplate.java:838) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:568) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:377) at hudson.plugins.ec2.EC2Cloud.provisionSlaveIfPossible(EC2Cloud.java:406) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:258) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) ... 69 more Caused by: java.lang.RuntimeException: Failed to serialize OfflineCause_groovyProxy#closureMap for class OfflineCause_groovyProxy at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 104 more Caused by: java.lang.RuntimeException: Failed to serialize groovy.lang.Closure#delegate for class Script1$_run_closure9 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:79) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 113 more Caused by: java.lang.RuntimeException: Failed to serialize groovy.lang.Script#binding for class Script1 at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 128 more Caused by: java.lang.RuntimeException: Failed to serialize groovy.lang.Binding#variables for class groovy.lang.Binding at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 137 more Caused by: java.lang.RuntimeException: Failed to serialize hudson.Launcher#listener for class hudson.Launcher$LocalLauncher at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:215) at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:183) at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138) at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:168) at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:109) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64) at com.thoughtworks.xstream.converters.collections.MapConverter.marshal(MapConverter.java:79) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 146 more Caused by: com.thoughtworks.xstream.converters.ConversionException: Could not call hudson.model.StreamBuildListener.writeObject() : Could not call hudson.remoting.RemoteOutputStream.writeObject() : null ---- Debugging information ---- message : Could not call hudson.remoting.RemoteOutputStream.writeObject() cause-exception : java.lang.NullPointerException cause-message : null ------------------------------- message : Could not call hudson.model.StreamBuildListener.writeObject() cause-exception : com.thoughtworks.xstream.converters.ConversionException cause-message : Could not call hudson.remoting.RemoteOutputStream.writeObject() : null ------------------------------- at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:141) at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doMarshal(SerializableConverter.java:259) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84) at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:224) at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:211) ... 161 more Caused by: com.thoughtworks.xstream.converters.ConversionException: Could not call hudson.remoting.RemoteOutputStream.writeObject() : null ---- Debugging information ---- message : Could not call hudson.remoting.RemoteOutputStream.writeObject() cause-exception : java.lang.NullPointerException cause-message : null ------------------------------- at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:141) at com.thoughtworks.xstream.converters.reflection.SerializableConverter.doMarshal(SerializableConverter.java:259) at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:83) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58) at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43) at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88) at com.thoughtworks.xstream.converters.reflection.SerializableConverter$1.writeToStream(SerializableConverter.java:140) at com.thoughtworks.xstream.core.util.CustomObjectOutputStream.writeObjectOverride(CustomObjectOutputStream.java:84) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:343) at hudson.util.StreamTaskListener.writeObject(StreamTaskListener.java:163) at sun.reflect.GeneratedMethodAccessor1916.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135) ... 168 more Caused by: java.lang.NullPointerException at hudson.remoting.RemoteOutputStream.writeObject(RemoteOutputStream.java:82) at sun.reflect.GeneratedMethodAccessor1917.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.thoughtworks.xstream.converters.reflection.SerializationMethodInvoker.callWriteObject(SerializationMethodInvoker.java:135) ... 182 more

Add Comment

[reese0809@gmail.com (JIRA)]

Charese Williams edited a comment on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

still happening :-( ***** if this helps a ec2-plugin spunned up node was left in the ec2 console but was not attached to Jenkins. this causes some type of lock. And then the orphaned node stays there and jenkins refuses to attach ANY other nodes, not even ones attached manually. The below stacktrace with the evil jenkins is shown below ****

[jenkins-ci.org@cowsgomoo.org (JIRA)]

Coltrey Mather commented on JENKINS-31804

Re: Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

For those of you who are terminating nodes manually, you can add something like this to your slave init script - it'll add a cron entry to shutdown if slave.jar isn't around. (crontab -l 2>/dev/null; echo "*/5 * * * * sleep 120 ; ps -ef|grep -v grep|grep slave.jar >&1 > /dev/null || sudo -n shutdown -h now") | crontab -

Add Comment

This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)

[fabrizio.manfredi@gmail.com (JIRA)]

FABRIZIO MANFREDI closed an issue as Cannot Reproduce

Jenkins / JENKINS-31804

Jenkins tells ec2 to spin up a node, aws spins up the node, but hudson/jenkins slave service never gets the memo and builds hang

Change By: FABRIZIO MANFREDI Status: Open Closed Resolution: Cannot Reproduce

Add Comment

This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)