[JIRA] (JENKINS-59900) Jenkins web interfece down

12 views
Skip to first unread message

arni.12@mail.ru (JIRA)

unread,
Oct 23, 2019, 6:23:05 AM10/23/19
to jenkinsc...@googlegroups.com
Владислав Ненашев created an issue
 
Jenkins / Bug JENKINS-59900
Jenkins web interfece down
Issue Type: Bug Bug
Assignee: Unassigned
Components: core
Created: 2019-10-23 10:22
Priority: Critical Critical
Reporter: Владислав Ненашев

Hi, 
we fixed the jenkins 2.176.3 web interface stop under rather strange circumstances
before stopping, multiple error messages preceded with for various tasks

 

Oct 18, 2019 9:55:21 AM hudson.model.RunMap retrieveWARNING: could not load /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68java.io.IOException: Unable to read /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68/build.xml                at hudson.XmlFile.unmarshal(XmlFile.java:181)                at hudson.XmlFile.unmarshal(XmlFile.java:161)                at org.jenkinsci.plugins.workflow.job.WorkflowRun.reload(WorkflowRun.java:723)                at hudson.model.Run.<init>(Run.java:330)                at org.jenkinsci.plugins.workflow.job.WorkflowRun.<init>(WorkflowRun.java:249)                at sun.reflect.GeneratedConstructorAccessor205.newInstance(Unknown Source)                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)                at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:166)                at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:143)                at hudson.model.RunMap.retrieve(RunMap.java:224)                at hudson.model.RunMap.retrieve(RunMap.java:57)                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)                at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)                at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)                at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74)                at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)                at java.util.AbstractMap$2$1.next(AbstractMap.java:418)                at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1186)                at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1300)                at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1297)                at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)                at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)                at java.util.PriorityQueue.offer(PriorityQueue.java:345)                at java.util.PriorityQueue.add(PriorityQueue.java:322)                at com.google.common.collect.Iterators$MergingIterator.computeNext(Iterators.java:1323)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at hudson.util.Iterators$9.fetch(Iterators.java:388)                at hudson.util.Iterators$9.hasNext(Iterators.java:375)                at hudson.util.Iterators$9.fetch(Iterators.java:388)                at hudson.util.Iterators$9.hasNext(Iterators.java:375)                at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at hudson.model.BuildTimelineWidget.doData(BuildTimelineWidget.java:64)                at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)                at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)                at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)                at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)                at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)                at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)                at org.kohsuke.stapler.Stapler.service(Stapler.java:238)                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)                at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)                at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)                at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)                at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)                at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)                at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)                at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)                at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)                at org.eclipse.jetty.server.Server.handle(Server.java:502)                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)                at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)                at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)                at java.lang.Thread.run(Thread.java:748)Caused by: com.thoughtworks.xstream.converters.ConversionException:  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)  :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)---- Debugging information ----message             :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)cause-exception     : com.thoughtworks.xstream.io.StreamExceptioncause-message       :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)class               : org.jenkinsci.plugins.workflow.job.WorkflowRunrequired-type       : java.util.HashMapconverter-type      : hudson.util.RobustReflectionConverterpath                : /flow-build/execution/timings/entry[4]line number         : 2813version             : not available-------------------------------                at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)                at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)                at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)                at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)                at hudson.util.XStream2.unmarshal(XStream2.java:161)                at hudson.util.XStream2.unmarshal(XStream2.java:132)                at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)                at hudson.XmlFile.unmarshal(XmlFile.java:178)                ... 144 moreCaused by: com.thoughtworks.xstream.io.StreamException:  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)                at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:124)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:141)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:118)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveUp(AbstractPullReader.java:113)                at com.thoughtworks.xstream.io.ReaderWrapper.moveUp(ReaderWrapper.java:40)                at com.thoughtworks.xstream.io.path.PathTrackingReader.moveUp(PathTrackingReader.java:42)                at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:364)                at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)                ... 154 moreCaused by: org.xmlpull.v1.XmlPullParserException: expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)                at org.kxml2.io.KXmlParser.exception(Unknown Source)                at org.kxml2.io.KXmlParser.error(Unknown Source)                at org.kxml2.io.KXmlParser.parseEndTag(Unknown Source)                at org.kxml2.io.KXmlParser.nextImpl(Unknown Source)                at org.kxml2.io.KXmlParser.next(Unknown Source)                at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)                ... 163 moreWARNING: Failed to retrieve incoming Pull Requests from Bitbucketcom.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .HttpResponseProxy{HTTP/1.1 401  [X-AREQUESTID: *134DEQFx597x100831134x4, X-ASEN: SEN-9068428, Cache-Control: no-cache, no-transform, Vary: X-AUSERNAME, Vary: X-AUSERID, Vary: Cookie, X-Content-Type-Options: nosniff, WWW-Authenticate: OAuth realm="https%3A%2F%2Fstash.delta.sbrf.ru", Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Date: Fri, 18 Oct 2019 06:57:10 GMT] ResponseEntityProxy{[Content-Type: application/json;charset=UTF-8,Chunked: true]}}                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:797)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:749)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getPullRequests(BitbucketServerAPIClient.java:339)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getIncomingOpenPullRequests(BitbucketServerAPIClient.java:334)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.loadPullRequests(NativeServerPushHookProcessor.java:254)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.getPullRequests(NativeServerPushHookProcessor.java:226)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.addPullRequests(NativeServerPushHookProcessor.java:186)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.heads(NativeServerPushHookProcessor.java:133)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerHeadEvent.heads(NativeServerHeadEvent.java:84)                at jenkins.scm.api.SCMHeadEvent.isMatch(SCMHeadEvent.java:137)                at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1406)                at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1184)                at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)                at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)                at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)                at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)                at java.util.concurrent.FutureTask.run(FutureTask.java:266)                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                at java.lang.Thread.run(Thread.java:748)

 
Add Comment Add Comment
 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo

arni.12@mail.ru (JIRA)

unread,
Oct 23, 2019, 6:25:04 AM10/23/19
to jenkinsc...@googlegroups.com
Hi, 
we fixed the jenkins 2.176.3 web interface stop under rather strange circumstances
before stopping, multiple error messages preceded with for various tasks
 
{code:java}

Oct 18, 2019 9:55:21 AM hudson.model.RunMap retrieveWARNING: could not load /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68java.io.IOException: Unable to read /opt/jenkins/jenkins_home/jobs/UCP/jobs/Release_UCP_BROKER/builds/68/build.xml                at hudson.XmlFile.unmarshal(XmlFile.java:181)                at hudson.XmlFile.unmarshal(XmlFile.java:161)                at org.jenkinsci.plugins.workflow.job.WorkflowRun.reload(WorkflowRun.java:723)                at hudson.model.Run.<init>(Run.java:330)                at org.jenkinsci.plugins.workflow.job.WorkflowRun.<init>(WorkflowRun.java:249)                at sun.reflect.GeneratedConstructorAccessor205.newInstance(Unknown Source)                at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)                at java.lang.reflect.Constructor.newInstance(Constructor.java:423)                at jenkins.model.lazy.LazyBuildMixIn.loadBuild(LazyBuildMixIn.java:166)                at jenkins.model.lazy.LazyBuildMixIn$1.create(LazyBuildMixIn.java:143)                at hudson.model.RunMap.retrieve(RunMap.java:224)                at hudson.model.RunMap.retrieve(RunMap.java:57)                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:501)                at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:483)                at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:381)                at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)                at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74)                at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)                at java.util.AbstractMap$2$1.next(AbstractMap.java:418)                at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1186)                at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1300)                at com.google.common.collect.Iterators$MergingIterator$1.compare(Iterators.java:1297)                at java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:670)                at java.util.PriorityQueue.siftUp(PriorityQueue.java:646)                at java.util.PriorityQueue.offer(PriorityQueue.java:345)                at java.util.PriorityQueue.add(PriorityQueue.java:322)                at com.google.common.collect.Iterators$MergingIterator.computeNext(Iterators.java:1323)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at hudson.util.Iterators$9.fetch(Iterators.java:388)                at hudson.util.Iterators$9.hasNext(Iterators.java:375)                at hudson.util.Iterators$9.fetch(Iterators.java:388)                at hudson.util.Iterators$9.hasNext(Iterators.java:375)                at com.google.common.collect.Iterators$7.computeNext(Iterators.java:647)                at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)                at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)                at hudson.model.BuildTimelineWidget.doData(BuildTimelineWidget.java:64)                at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)                at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)                at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)                at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)                at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)                at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:535)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:456)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:219)                at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)                at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:747)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:878)                at org.kohsuke.stapler.Stapler.invoke(Stapler.java:676)                at org.kohsuke.stapler.Stapler.service(Stapler.java:238)                at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)                at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)                at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:246)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.locale.LocaleFilter.doFilter(LocaleFilter.java:42)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)                at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)                at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)                at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:114)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:92)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:128)                at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)                at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)                at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)                at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)                at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)                at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)                at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)                at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)                at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)                at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)                at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1701)                at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)                at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)                at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)                at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1668)                at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)                at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)                at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)                at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)                at org.eclipse.jetty.server.Server.handle(Server.java:502)                at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)                at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)                at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)                at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)                at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)                at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)                at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)                at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)                at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)                at java.lang.Thread.run(Thread.java:748)Caused by: com.thoughtworks.xstream.converters.ConversionException:  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)  :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)---- Debugging information ----message             :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)cause-exception     : com.thoughtworks.xstream.io.StreamExceptioncause-message       :  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)class               : org.jenkinsci.plugins.workflow.job.WorkflowRunrequired-type       : java.util.HashMapconverter-type      : hudson.util.RobustReflectionConverterpath                : /flow-build/execution/timings/entry[4]line number         : 2813version             : not available-------------------------------                at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)                at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)                at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)                at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)                at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)                at hudson.util.XStream2.unmarshal(XStream2.java:161)                at hudson.util.XStream2.unmarshal(XStream2.java:132)                at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)                at hudson.XmlFile.unmarshal(XmlFile.java:178)                ... 144 moreCaused by: com.thoughtworks.xstream.io.StreamException:  : expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)                at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:124)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:141)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:118)                at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveUp(AbstractPullReader.java:113)                at com.thoughtworks.xstream.io.ReaderWrapper.moveUp(ReaderWrapper.java:40)                at com.thoughtworks.xstream.io.path.PathTrackingReader.moveUp(PathTrackingReader.java:42)                at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:364)                at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:268)                at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)                ... 154 moreCaused by: org.xmlpull.v1.XmlPullParserException: expected: /entry read: iota (position:END_TAG </iota>@2813:21 in java.io.InputStreamReader@b188934)                at org.kxml2.io.KXmlParser.exception(Unknown Source)                at org.kxml2.io.KXmlParser.error(Unknown Source)                at org.kxml2.io.KXmlParser.parseEndTag(Unknown Source)                at org.kxml2.io.KXmlParser.nextImpl(Unknown Source)                at org.kxml2.io.KXmlParser.next(Unknown Source)                at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)                ... 163 moreWARNING: Failed to retrieve incoming Pull Requests from Bitbucketcom.cloudbees.jenkins.plugins.bitbucket.api.BitbucketRequestException: HTTP request error. Status: 401: .HttpResponseProxy{HTTP/1.1 401  [X-AREQUESTID: *134DEQFx597x100831134x4, X-ASEN: SEN-9068428, Cache-Control: no-cache, no-transform, Vary: X-AUSERNAME, Vary: X-AUSERID, Vary: Cookie, X-Content-Type-Options: nosniff, WWW-Authenticate: OAuth realm="https%3A%2F%2Fstash.delta.sbrf.ru", Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Date: Fri, 18 Oct 2019 06:57:10 GMT] ResponseEntityProxy{[Content-Type: application/json;charset=UTF-8,Chunked: true]}}                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getRequest(BitbucketServerAPIClient.java:797)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getResources(BitbucketServerAPIClient.java:749)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getPullRequests(BitbucketServerAPIClient.java:339)                at com.cloudbees.jenkins.plugins.bitbucket.server.client.BitbucketServerAPIClient.getIncomingOpenPullRequests(BitbucketServerAPIClient.java:334)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.loadPullRequests(NativeServerPushHookProcessor.java:254)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.getPullRequests(NativeServerPushHookProcessor.java:226)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.addPullRequests(NativeServerPushHookProcessor.java:186)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerPushHookProcessor$HeadEvent.heads(NativeServerPushHookProcessor.java:133)                at com.cloudbees.jenkins.plugins.bitbucket.hooks.NativeServerHeadEvent.heads(NativeServerHeadEvent.java:84)                at jenkins.scm.api.SCMHeadEvent.isMatch(SCMHeadEvent.java:137)                at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.processHeadUpdate(MultiBranchProject.java:1406)                at jenkins.branch.MultiBranchProject$SCMEventListenerImpl.onSCMHeadEvent(MultiBranchProject.java:1184)                at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:246)                at jenkins.scm.api.SCMHeadEvent$DispatcherImpl.fire(SCMHeadEvent.java:229)                at jenkins.scm.api.SCMEvent$Dispatcher.run(SCMEvent.java:505)                at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:58)                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)                at java.util.concurrent.FutureTask.run(FutureTask.java:266)                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)                at java.lang.Thread.run(Thread.java:748)

{code}
  Further, an interesting message was found in the log, please pay attention to the message time
{code:java}
Oct 18, 2019 9:57:26 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher logLastLinesSEVERE: Error in provisioning; agent=KubernetesSlave name: massive-8g-os-bmb0v, template=PodTemplate{inheritFrom='', name='massive-8G-os', namespace='', label='openshift_8Gb DefaultPool Linux_Default ansible linux masterLin oair rhel7', nodeSelector='', nodeUsageMode=EXCLUSIVE, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], containers=[ContainerTemplate{name='jnlp', image='sbt-gitlab.delta.sbrf.ru:4567/oair/agent-docker:1.28_os', workingDir='/u01/jenkins', command='', args='-noreconnect ${computer.jnlpmac} ${computer.name}', resourceRequestCpu='1', resourceRequestMemory='4Gi', resourceLimitCpu='2', resourceLimitMemory='8Gi', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@636583a}]}. Container jnlp. Logs: INFO: Using Remoting version: 3.29Oct 18, 2019 9:56:22 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDirINFO: Using /u01/jenkins/remoting as a remoting work directoryBoth error and output logs will be printed to /u01/jenkins/remotingOct 18, 2019 9:56:23 AM hudson.remoting.jnlp.Main$CuiListener statusINFO: Locating server among [https://sbt-jenkins.ca.sbrf.ru/jenkins/]Oct 18, 2019 9:57:24 AM hudson.remoting.jnlp.Main$CuiListener error
{code}

arni.12@mail.ru (JIRA)

unread,
Oct 23, 2019, 6:27:03 AM10/23/19
to jenkinsc...@googlegroups.com

Around this time, the web interface stopped responding, connections hung in anticipation, until they fell timeout

Before restarting, I prepared a jstack, but I did not find the reason for the lock in it

 

arni.12@mail.ru (JIRA)

unread,
Oct 23, 2019, 6:53:02 AM10/23/19
to jenkinsc...@googlegroups.com
Change By: Владислав Ненашев
Attachment: jstackjava2
Attachment: jstackjava1

arni.12@mail.ru (JIRA)

unread,
Oct 31, 2019, 7:23:03 AM10/31/19
to jenkinsc...@googlegroups.com

arni.12@mail.ru (JIRA)

unread,
Nov 18, 2019, 11:22:06 PM11/18/19
to jenkinsc...@googlegroups.com
Change By: Владислав Ненашев
Attachment: Legacy_jstackjava1

arni.12@mail.ru (JIRA)

unread,
Nov 18, 2019, 11:22:06 PM11/18/19
to jenkinsc...@googlegroups.com
Владислав Ненашев commented on Bug JENKINS-59900
 
Re: Jenkins web interfece down

I got a similar problem on another instance[^Legacy_jstackjava1]
Reply all
Reply to author
Forward
0 new messages