Crashes with LTS 2.289.1

28 views
Skip to first unread message

Alan Sparks

unread,
Jun 3, 2021, 11:33:33 AMJun 3
to jenkins...@googlegroups.com

Has anyone been having random crashes now with new 2.289.1 LTS?  Running successfully the last LTS (2.277.4), and upgraded to new LTS and upgraded plugins.

 

Now it’s crashing at various times, usually while running jobs.   It’s an EC2 instance, so I’ve lost some info when autoscaling kicks it out, but while watching today I notice the LA of the instance hit 43, and kswapd was the 100% CPU user.  The build slave it was running was OK, and still running steps.

 

I saw this in the logs before it hung:

 

2021-06-03 15:20:04.725+0000 [id=744]   WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[Owner[AMI Builds/Foundation AMI Build/centos-7/br1/1:AMI Builds/Foundation AMI Build/centos-7/br1 #1]]

java.util.concurrent.TimeoutException: Timeout waiting for task.

        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:259)

        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)

        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:950)

        at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:398)

        at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:401)

        at hudson.model.Executor.isDisplayCell(Executor.java:668)

        at hudson.model.Computer.getDisplayExecutors(Computer.java:1011)

        at sun.reflect.GeneratedMethodAccessor578.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:498)

        at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)

        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)

        at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)

        at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)

        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)

        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)

        at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)

        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)

        at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)

        at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:100)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:99)

        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)

        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)

        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)

        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)

        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:95)

        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:64)

        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:54)

        at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)

        at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)

        at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:240)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:834)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)

        at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)

        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)

        at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)

        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)

        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)

        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)

        at org.kohsuke.stapler.Stapler.service(Stapler.java:240)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)

        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)

        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)

        at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)

        at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)

        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)

        at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:64)

        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)

        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:92)

        at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)

        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:105)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)

        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:92)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:218)

        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:212)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)

        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)

        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:62)

        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:97)

        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:109)

        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:168)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)

        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)

        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)

        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)

        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)

        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)

        at org.eclipse.jetty.server.Server.handle(Server.java:516)

        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)

        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)

        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)

        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:279)

        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)

        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)

        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)

        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)

        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)

        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:882)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1036)

        at java.lang.Thread.run(Thread.java:748)

2021-06-03 15:20:04.877+0000 [id=84]    INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#24]: checking /tmp/jenkins-e734755e/workspace/MI_Build_ubuntu-20.04LTS_alstest on i-01b582bfd074412a7 / waiting for i-01b582bfd074412a7 id=1546 unresponsive for 9.9 sec

2021-06-03 15:20:04.942+0000 [id=84]    INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#35]: checking /tmp/jenkins-e734755e/workspace/oundation_AMI_Build_centos-7_br1 on i-01b582bfd074412a7 / waiting for i-01b582bfd074412a7 id=1547 unresponsive for 29 sec

2021-06-03 15:21:34.010+0000 [id=84]    INFO    o.j.p.w.s.concurrent.Timeout#lambda$ping$0: org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep [#23]: checking /tmp/jenkins-e734755e/workspace/oundation_AMI_Build_centos-7_br1 on i-01b582bfd074412a7 / waiting for i-01b582bfd074412a7 id=1551 unresponsive for 57 sec

2021-06-03 15:21:40.862+0000 [id=35]    INFO    c.a.j.e.EC2RetentionStrategy#check: Check if node idle i-01b582bfd074412a7

2021-06-03 15:21:34.779+0000 [id=2183]  INFO    hudson.model.AsyncPeriodicWork#lambda$doRun$0: Started DockerContainerWatchdog Asynchronous Periodic Work

2021-06-03 15:21:49.051+0000 [id=39]    INFO    c.a.j.e.EC2RetentionStrategy#check: Check if node idle i-01b582bfd074412a7

2021-06-03 15:21:42.953+0000 [id=40]    INFO    j.m.api.Metrics$HealthChecker#doRun: jenkins.metrics.api.Metrics$HealthChecker thread is still running. Execution aborted.

2021-06-03 15:22:18.182+0000 [id=744]   WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[Owner[AMI Builds/Foundation AMI Build/centos-7/br1/1:AMI Builds/Foundation AMI Build/centos-7/br1 #1]]

 

Reply all
Reply to author
Forward
0 new messages