slave termination error

[Ashish Yadav]

I am running Jenkins v1.609.3

The jobs on my slave nodes are executing correctly. However, now I am getting the following error for every one of my nodes.

Executor #0 for NodeABC : executing job1 #104 termination trace
hudson.model.Computer$TerminationRequest: Termination requested at Wed Jan 27 21:46:03 CST 2016 by Thread[Monitoring thread for Response Time started on Wed Jan 27 21:46:03 CST 2016,5,main] [id=278206]
	at hudson.model.Computer.recordTermination(Computer.java:216)
	at hudson.model.Computer.disconnect(Computer.java:466)
	at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:601)
	at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:72)
	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:302)

All my slaves are online and connected to master. Do I need to disconnect and re-connect them?

[Ashish Yadav]

I restarted the slave node and am now getting this error even though the job finishes successfully.

Jan 28, 2016 8:04:30 AM WARNING hudson.model.Executor finish2

Executor #0 for NodeABC : executing dummy-job #49 termination trace
hudson.model.Computer$TerminationRequest: Termination requested at Wed Jan 27 21:46:03 CST 2016 by Thread[Monitoring thread for Response Time started on Wed Jan 27 21:46:03 CST 2016,5,main] [id=278206]
	at hudson.model.Computer.recordTermination(Computer.java:216)
	at hudson.model.Computer.disconnect(Computer.java:466)
	at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:601)
	at hudson.node_monitors.ResponseTimeMonitor$1.monitor(ResponseTimeMonitor.java:72)
	at hudson.node_monitors.AbstractNodeMonitorDescriptor$Record.run(AbstractNodeMonitorDescriptor.java:302)

Jan 28, 2016 8:04:30 AM WARNING hudson.model.Executor finish2

Executor #0 for NodeABC : executing dummy-job #49 termination trace
hudson.model.Computer$TerminationRequest: Termination requested at Thu Jan 28 07:10:51 CST 2016 by Thread[Handling POST /computer/NodeABC/doDisconnect from xxx.xxx.xxx.xxx : RequestHandlerThread[#47],5,main] [id=273178] from HTTP request for http://xxx/computer/NodeABC/doDisconnect
	at hudson.model.Computer.recordTermination(Computer.java:207)
	at hudson.model.Computer.disconnect(Computer.java:466)
	at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:601)
	at hudson.slaves.SlaveComputer.doDoDisconnect(SlaveComputer.java:594)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
	at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:211)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Jan 28, 2016 8:04:30 AM WARNING hudson.model.Executor finish2

Executor #0 for NodeABC : executing dummy-job-to-test-success-email #49 termination trace
hudson.model.Computer$TerminationRequest: Termination requested at Thu Jan 28 08:01:39 CST 2016 by Thread[Handling POST /computer/NodeABC/doDisconnect from xxx.xxx.xxx.xxx : RequestHandlerThread[#42],5,main] [id=272809] from HTTP request for http://xxx/computer/NodeABC/doDisconnect
	at hudson.model.Computer.recordTermination(Computer.java:207)
	at hudson.model.Computer.disconnect(Computer.java:466)
	at hudson.slaves.SlaveComputer.disconnect(SlaveComputer.java:601)
	at hudson.slaves.SlaveComputer.doDoDisconnect(SlaveComputer.java:594)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298)	at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:46)
	at org.kohsuke.stapler.Function$InterceptedFunction.invoke(Function.java:399)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96)
	at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$13.dispatch(MetaClass.java:411)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:211)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123)
	at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:95)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:370)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
	at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
	at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/D2CF6DE0.151AD%25ashish.yadav%40firemon.com.
For more options, visit https://groups.google.com/d/optout.

[Ashish Yadav]

Any guidance is greatly appreciated…

[Baptiste Mathus]

As you can see in the stack trace, this error seems to come from the ResponseTimeMonitor. Are per chance your master node and the build agents not on the same network?
Could you try pinging nodes from the master and paste the result here so that a can judge if it appears normal or not?

My 2 cents

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/D2CF88FE.151D5%25ashish.yadav%40firemon.com.

[Ashish Yadav]

Thanks Mathus. We were able to resolve the issue by restarting the master. Here’s what happened. A network outage caused the master to loose connectivity to the slaves. After some time, the master issues a termination request to the slaves. When the network issue was resolved, the slaves reconnected. However, the termination request was never cancelled by the master.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS7haH8WbcwXNDiSynsn6eWsme%2BY_LV8UqTcCHvMMFDKxA%40mail.gmail.com.