[JIRA] (JENKINS-14362) 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

[chris+jira@aptivate.org (JIRA)]

Chris Wilson created JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException Issue Type: Bug Affects Versions: current Assignee: Unassigned Components: core Created: 09/Jul/12 4:31 PM Description: Jenkins starts using 100% CPU after a few days. Using jstack I see several threads trying to write compressed output, and apparently not changing over time: java.util.zip.Deflater.deflate(byte[], int, int) @bci=55, line=322 (Compiled frame; information may be imprecise) java.util.zip.DeflaterOutputStream.deflate() @bci=14, line=176 (Compiled frame) java.util.zip.DeflaterOutputStream.write(byte[], int, int) @bci=108, line=135 (Compiled frame) java.util.zip.GZIPOutputStream.write(byte[], int, int) @bci=4, line=89 (Compiled frame) org.kohsuke.stapler.compression.FilterServletOutputStream.write(byte[], int, int) @bci=7, line=31 (Compiled frame) sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=220 (Interpreted frame) sun.nio.cs.StreamEncoder.implClose() @bci=84, line=315 (Interpreted frame) sun.nio.cs.StreamEncoder.close() @bci=18, line=148 (Interpreted frame) java.io.OutputStreamWriter.close() @bci=4, line=233 (Interpreted frame) java.io.PrintWriter.close() @bci=21, line=312 (Interpreted frame) org.kohsuke.stapler.compression.CompressionFilter.reportException(java.lang.Exception, javax.servlet.http.HttpServletResponse) @bci=112, line=77 (Interpreted frame) org.kohsuke.stapler.compression.CompressionFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=57, line=53 (Compiled frame) winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame) winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame) hudson.util.CharacterEncodingFilter.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=43, line=81 (Compiled frame) winstone.FilterConfiguration.execute(javax.servlet.ServletRequest, javax.servlet.ServletResponse, javax.servlet.FilterChain) @bci=25, line=194 (Compiled frame) winstone.RequestDispatcher.doFilter(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=48, line=366 (Compiled frame) winstone.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse) @bci=483, line=331 (Compiled frame) winstone.RequestHandlerThread.processRequest(winstone.WebAppConfiguration, winstone.WinstoneRequest, winstone.WinstoneResponse, java.lang.String) @bci=38, line=215 (Compiled frame) winstone.RequestHandlerThread.run() @bci=631, line=138 (Compiled frame) java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame) java.util.concurrent.FutureTask$Sync.innerRun() @bci=29, line=334 (Interpreted frame) java.util.concurrent.FutureTask.run() @bci=4, line=166 (Interpreted frame) winstone.BoundedExecutorService$1.run() @bci=4, line=77 (Compiled frame) java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=46, line=1110 (Compiled frame) java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=603 (Interpreted frame) java.lang.Thread.run() @bci=11, line=679 (Interpreted frame) I'm suspecting but not 100% sure that these threads are in an infinite loop (livelocked). I'm struggling to see what other threads might be doing this. This JVM was not started with debugging enabled to attach a debugger for analysis. I've enabled it now. Stack traces attached as files below. Environment: [chris@fen-vz-jenkins ~]$ java -version java version "1.6.0_22" OpenJDK Runtime Environment (IcedTea6 1.10.8) (rhel-1.27.1.10.8.el5_8-i386) OpenJDK Server VM (build 20.0-b11, mixed mode) [chris@fen-vz-jenkins ~]$ uname -a Linux fen-vz-jenkins.fen.aptivate.org 2.6.32-7-pve #1 SMP Mon Feb 13 07:33:21 CET 2012 i686 i686 i386 GNU/Linux [chris@fen-vz-jenkins ~]$ cat /etc/issue CentOS release 5.8 (Final) Project: Jenkins Priority: Major Reporter: Chris Wilson

This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira

[chris+jira@aptivate.org (JIRA)]

Chris Wilson updated JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Change By: Chris Wilson (09/Jul/12 4:33 PM) Attachment: jenkins.stacktrace.1 Attachment: jenkins.stacktrace.2 Attachment: jenkins.stacktrace.3

[david.parsson@gmail.com (JIRA)]

David Pärsson commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have seen this issue (or a very similar one) when trying to generate Performance reports for huge JMeter reports. Jenkins v1.471 and Performance plugin v1.8.

[david.parsson@gmail.com (JIRA)]

David Pärsson edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I think I have seen this issue (or a very similar one) when trying to generate Performance reports for huge JMeter reports. Jenkins v1.471 and Performance plugin v1.8.

[robert_lloyd_1981@yahoo.com (JIRA)]

Bob Lloyd commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have the same issue, but cannot track it to a specific job. I am not generating Performance reports (to my knowledge, unless it's happening w/o my intention). I've attached my log below (though, it's pretty much the same as above). This happens for me after about 36 hours of up-time for Jenkins. I have ~30 jobs running on 6 servers. One server runs jobs pretty much constantly, while other servers run much less frequently. "RequestHandlerThread541" daemon prio=6 tid=0x4959d800 nid=0x8c4 runnable [0x4b40f000] java.lang.Thread.State: RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Unknown Source) locked <0x19974c70> (a java.util.zip.ZStreamRef) at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) at java.util.zip.DeflaterOutputStream.write(Unknown Source) at java.util.zip.GZIPOutputStream.write(Unknown Source) locked <0x19974c80> (a java.util.zip.GZIPOutputStream) at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source) at sun.nio.cs.StreamEncoder.implClose(Unknown Source) at sun.nio.cs.StreamEncoder.close(Unknown Source) locked <0x19976ce0> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.close(Unknown Source) at java.io.PrintWriter.close(Unknown Source) locked <0x19976ce0> (a java.io.OutputStreamWriter) at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Locked ownable synchronizers: <0x19973bf8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

[robert_lloyd_1981@yahoo.com (JIRA)]

Bob Lloyd edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have the same issue, but cannot track it to a specific job. I am not generating Performance reports (to my knowledge, unless it's happening w/o my intention). I've attached my log below (though, it's pretty much the same as above). I'm running with Jenkins 1.491 with Sun JDK 1.6.0_26

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I still have this problem occurring at least twice a week causing cpu at 400% (on a 4 CPUs machine) that forces me to restart the server. When it happens I can workaround it (temporarily) by using the 'monitoring' plugin, going to the list of threads and sort the table by execution time and kill the threads actively running "DeflaterOutputStream.deflate" that are on top of the table. This makes CPU drop back to normal values but eventually a restart will be needed at the end of the day. Here is a stack trace collected from our logs: WARNING: Untrapped servlet exception winstone.ClientSocketException: Failed to write to client at winstone.ClientOutputStream.write(ClientOutputStream.java:41) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:181) at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:119) at winstone.WinstoneOutputStream.write(WinstoneOutputStream.java:112) at java.util.zip.GZIPOutputStream.finish(GZIPOutputStream.java:169) at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:238) at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36) at net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:46) at java.io.FilterOutputStream.close(FilterOutputStream.java:160) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:320) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.BufferedWriter.close(BufferedWriter.java:266) at org.dom4j.io.XMLWriter.close(XMLWriter.java:286) at org.kohsuke.stapler.jelly.HTMLWriterOutput.close(HTMLWriterOutput.java:70) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:56) at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107) at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:563) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:384) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:574) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:659) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:488) at org.kohsuke.stapler.Stapler.service(Stapler.java:162) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at winstone.ClientOutputStream.write(ClientOutputStream.java:39) ... 88 more

[bruce.edge@gmail.com (JIRA)]

Bruce Edge commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have the same problem. It triggers soft lockups in the kernel of the jenkins host: 901 Jan 30 02:31:58 build kernel: [589969.228931] INFO: rcu_sched detected stall on CPU 3 (t=57719 jiffies) 902 Jan 30 02:31:58 build kernel: [589969.228939] INFO: rcu_sched detected stall on CPU 2 (t=57719 jiffies) 903 Jan 30 02:31:58 build kernel: [589969.228929] INFO: rcu_sched detected stall on CPU 1 (t=57719 jiffies) 904 Jan 30 02:31:58 build kernel: [589969.228944] INFO: rcu_sched detected stall on CPU 4 (t=57719 jiffies) 905 Jan 30 02:31:58 build kernel: [589969.228939] sending NMI to all CPUs: Note the timestamp correlation between the exception below and the stalls above. 712 Jan 30, 2013 2:31:58 AM org.kohsuke.stapler.compression.CompressionFilter reportException 6713 WARNING: Untrapped servlet exception 6714 winstone.ClientSocketException: Failed to write to client 6715 at winstone.ClientOutputStream.write(ClientOutputStream.java:41) 6716 at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) 6717 at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) 6718 at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:165) 6719 at winstone.WinstoneOutputStream.flush(WinstoneOutputStream.java:217) 6720 at winstone.WinstoneOutputStream.close(WinstoneOutputStream.java:227) 6721 at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:241) 6722 at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36) 6723 at net.bull.javamelody.FilterServletOutputStream.close(FilterServletOutputStream.java:46) This happens most nights at the same time.

[ricko2000@att.net (JIRA)]

Richard Otter commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

On my system, just upgraded to 1.492 from the last LTS 1.480.1, this happens whenever I display an EMMA code coverage report. The server java process takes upwards of 80% CPU for at least 20 min until I use the method above to kill the "Active Request" in the Monitoring plugin. I've seen this behavior for months, but this jira thread shows me how to kill the thread. That's progress.

[jan.hoppe@heidelberg.com (JIRA)]

Jan Hoppe commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Hi, we have been running into these troubles often the last time. I found a "disable" flag for "reportException". Add -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true to your VM-Options in jenkins xml! Good luck, Jan

[josesa@java.net (JIRA)]

Jose Sa commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I've added that entry to my startup script 2 weeks ago and also haven't experienced 100% cpu symptom since then.

[uwe+jenkins@bsdx.de (JIRA)]

Uwe Stuehler commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Could this be related to the thread-safety (or not) of java.util.zip? https://forums.oracle.com/forums/thread.jspa?messageID=4627097 In GNU Classpath, java.util.zip.Deflater is clearly documented as not thread-safe due to limitations of the API. I couldn't find a mention of the thread-safety issue in Sun/Oracle documentation, but some have concluded that their implementation isn't thread-safe either.

[reynald.borer@gmail.com (JIRA)]

Reynald Borer commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Hi, I'm also experiencing this issue with Jenkins version 1.506 on the following environment: Jenkins 1.506 Debian Squeeze 64 bits (Linux buildsrv1 3.2.0-0.bpo.2-amd64) Java(TM) SE Runtime Environment (build 1.6.0_37-b06)

[sini.makela@f-secure.com (JIRA)]

Sini Mäkelä commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

We are having this same issue with Jenkins v1.510. The workaround suggested from Jan Hoppe seem to have solved the problem also in our environment. After adding "JAVA_ARGS=-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" into /etc/default/jenkins the CPU usage has normalized based on a couple of days experience.

[jan.hoppe@heidelberg.com (JIRA)]

Jan Hoppe commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I blogged a little bit about detecting that problem: http://devophuman.blogspot.com/2013/04/jenkins-is-going-wild.html

[catalan@adobe.com (JIRA)]

david catalan commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Hello, we are experiencing the same issue on our 2 Jenkins instances. I have found a way to reproduce the issue: 1. Create a job which generates a big console output (minimum 25Mb) (ie. a shell action doing a cat of a big text file) 2. On your screen, keep an eye on the jenkins log (file tail) + the cpu utilization of the jenkins process (top command or visualVM) 3. On Jenkins interface, navigate to the full console output of this job (http://<JENKINS_HOST>/job/<JENKINS_JOB/lastBuild/consoleFull) 4. Keep refreshing the console page without letting the page load completly => You should see the "java.util.zip.Deflater.deflate" stack trace in the the jenkins.log 5. Keep refreshing until the Jenkins process uses all the CPU(s)!! Tested on our 2 instances and, depending on the box current use + the size of the console, the CPU issue takes more or less time to occur. Finally, after applying the "-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" fix, the exception still occurs in the log BUT, no more CPU overload!! I see that a lot of users are experiencing this issue on multiple configuration (OS/Jenkins Version)!! Also a lot of users are spending time on it trying to debug and find out the reason/solution of this issue. Now, I would like to see some reaction from the Jenkins team. Are they taking care of this issue? We expect some solution from their side. Thanks in advance, David.

[kacynski.walt@aoins.com (JIRA)]

Walter Kacynski commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have tried option -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true however, the Content-Encoding header that is returned is always of type gzip. Does it matter where in the arguments section that this needs to be set? After looking a little more at the code it seems that the disable doesn't actually disable to compression just the error reporting? Why would this be?

[scott.oster@inventrio.com (JIRA)]

Scott Oster commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I have this problem as well with jenkins version 1.509. Using the JavaMelody dashboard to kill the threads stuck in ava.util.zip.DeflaterOutputStream.deflate reduced the CPU usage (without seeming to cause problems).

[corey@ooyala.com (JIRA)]

Corey O'Connor commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

We are using jenkins 1.484 and are experiencing the same issue. The "-Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true" option did result in the threads no longer taking 100% CPU. However, this broke loading the CSS. Perhaps due to what Walter Kacynski noticed.

[guy.kisel@gmail.com (JIRA)]

Guy Kisel commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

On 1.511 this is still happening, even though I set: <arguments>-Xrs -Xmx1024m -XX:MaxPermSize=512m -Dhudson.lifecycle=hudson.lifecycle.WindowsServiceLifecycle -jar "%BASE%\jenkins.war" --httpPort=8081 --prefix=/jenkins -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true</arguments>

[corey@ooyala.com (JIRA)]

Corey O'Connor commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Since my previous comment I have noticed two incidents of 100% cpu usage from a zip compression thread that appears "stuck". Either org.kohsuke.stapler.compression.CompressionFilter.disabled=true reduces the frequency ; a different path results in the same behavior ; or the reduction in frequency is all by chance.

[jan.hoppe@heidelberg.com (JIRA)]

Jan Hoppe commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

You should create a thread dump from your vm to enrich this report. Configuration is mentioned in http://devophuman.blogspot.com/2013/04/jenkins-is-going-wild.html. The VisualVM is part of Oracles JDK...

[slowbyte+jenkins@gmail.com (JIRA)]

Jaen S commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Experiencing this daily as well. Fairly frustrating. Jenkins 1.514 java version "1.6.0_27" OpenJDK Runtime Environment (IcedTea6 1.12.3) (6b27-1.12.3-0ubuntu1~12.04.1) OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode) Stack trace of looping thread: java.lang.Thread.State: RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:322) - locked <0x000000008cbec4d8> (a java.util.zip.ZStreamRef) at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:176) at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:135) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:89) - locked <0x000000008cbec490> (a java.util.zip.GZIPOutputStream) at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:315) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:148) - locked <0x000000008cc1a478> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.PrintWriter.close(PrintWriter.java:312) - locked <0x000000008cc1a478> (a java.io.OutputStreamWriter) at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)

[guy.kisel@gmail.com (JIRA)]

Guy Kisel commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Thread dump of this issue occuring on 1.516: Thread Dump AWT-Windows "AWT-Windows" Id=217 Group=main RUNNABLE (in native) at sun.awt.windows.WToolkit.eventLoop(Native Method) at sun.awt.windows.WToolkit.run(WToolkit.java:299) at java.lang.Thread.run(Thread.java:722) bot1-input "bot1-input" Id=167 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) locked java.lang.Object@362d4a04 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) locked sun.security.ssl.AppInputStream@57566865 at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) locked java.io.InputStreamReader@69dfebae at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:154) at java.io.BufferedReader.readLine(BufferedReader.java:317) locked java.io.InputStreamReader@69dfebae at java.io.BufferedReader.readLine(BufferedReader.java:382) at org.pircbotx.InputThread.run(InputThread.java:83) bot1-output "bot1-output" Id=168 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.pircbotx.OutputThread.run(OutputThread.java:101) Channel reader thread: serial-server-1 "Channel reader thread: serial-server-1" Id=146 Group=main WAITING on java.lang.Object@61e52512 at java.lang.Object.wait(Native Method) waiting on java.lang.Object@61e52512 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: ubuntu-test-VM-3 "Channel reader thread: ubuntu-test-VM-3" Id=148 Group=main WAITING on java.lang.Object@1c5e0a0a at java.lang.Object.wait(Native Method) waiting on java.lang.Object@1c5e0a0a at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) ConnectorThread:[http-8081] "ConnectorThread:[http-8081]" Id=12 Group=main RUNNABLE (in native) at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:125) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) locked java.net.SocksSocketImpl@773d4576 at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at winstone.HttpListener.run(HttpListener.java:139) at java.lang.Thread.run(Thread.java:722) DestroyJavaVM "DestroyJavaVM" Id=15 Group=main RUNNABLE Executor #0 for D0-Debian-Slave-1 "Executor #0 for D0-Debian-Slave-1" Id=86 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D0-Debian-Slave-2 "Executor #0 for D0-Debian-Slave-2" Id=91 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D1-debian-FreshInstall-1 "Executor #0 for D1-debian-FreshInstall-1" Id=116 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-1 "Executor #0 for D2-Debian-Slave-1" Id=111 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-2 "Executor #0 for D2-Debian-Slave-2" Id=106 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-1 "Executor #0 for D3-Debian-Slave-1" Id=96 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-2 "Executor #0 for D3-Debian-Slave-2" Id=101 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for master "Executor #0 for master" Id=38 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for serial-server-1 "Executor #0 for serial-server-1" Id=48 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-1 "Executor #0 for ubuntu-test-VM-1" Id=64 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-2 "Executor #0 for ubuntu-test-VM-2" Id=75 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-3 "Executor #0 for ubuntu-test-VM-3" Id=53 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-1 "Executor #1 for D0-Debian-Slave-1" Id=87 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-2 "Executor #1 for D0-Debian-Slave-2" Id=92 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-1 "Executor #1 for D2-Debian-Slave-1" Id=112 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-2 "Executor #1 for D2-Debian-Slave-2" Id=107 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-1 "Executor #1 for D3-Debian-Slave-1" Id=97 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-2 "Executor #1 for D3-Debian-Slave-2" Id=102 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for master "Executor #1 for master" Id=39 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for serial-server-1 "Executor #1 for serial-server-1" Id=49 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-1 "Executor #1 for ubuntu-test-VM-1" Id=65 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-2 "Executor #1 for ubuntu-test-VM-2" Id=76 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-3 "Executor #1 for ubuntu-test-VM-3" Id=54 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-1 "Executor #2 for D0-Debian-Slave-1" Id=88 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-2 "Executor #2 for D0-Debian-Slave-2" Id=93 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-1 "Executor #2 for D2-Debian-Slave-1" Id=113 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-2 "Executor #2 for D2-Debian-Slave-2" Id=108 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-1 "Executor #2 for D3-Debian-Slave-1" Id=98 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-2 "Executor #2 for D3-Debian-Slave-2" Id=103 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for master "Executor #2 for master" Id=40 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for serial-server-1 "Executor #2 for serial-server-1" Id=50 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-1 "Executor #2 for ubuntu-test-VM-1" Id=66 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-2 "Executor #2 for ubuntu-test-VM-2" Id=77 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-3 "Executor #2 for ubuntu-test-VM-3" Id=55 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-1 "Executor #3 for D0-Debian-Slave-1" Id=89 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-2 "Executor #3 for D0-Debian-Slave-2" Id=94 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-1 "Executor #3 for D2-Debian-Slave-1" Id=114 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-2 "Executor #3 for D2-Debian-Slave-2" Id=109 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-1 "Executor #3 for D3-Debian-Slave-1" Id=99 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-2 "Executor #3 for D3-Debian-Slave-2" Id=104 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for master "Executor #3 for master" Id=41 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for serial-server-1 "Executor #3 for serial-server-1" Id=51 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-1 "Executor #3 for ubuntu-test-VM-1" Id=67 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-2 "Executor #3 for ubuntu-test-VM-2" Id=78 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-3 "Executor #3 for ubuntu-test-VM-3" Id=56 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-1 "Executor #4 for D0-Debian-Slave-1" Id=90 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-2 "Executor #4 for D0-Debian-Slave-2" Id=95 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-1 "Executor #4 for D2-Debian-Slave-1" Id=115 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-2 "Executor #4 for D2-Debian-Slave-2" Id=110 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-1 "Executor #4 for D3-Debian-Slave-1" Id=100 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-2 "Executor #4 for D3-Debian-Slave-2" Id=105 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for master "Executor #4 for master" Id=42 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-1 "Executor #4 for ubuntu-test-VM-1" Id=68 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-2 "Executor #4 for ubuntu-test-VM-2" Id=79 Group=main TIMED_WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at hudson.util.OneShotEvent.block(OneShotEvent.java:86) at hudson.model.Queue.pop(Queue.java:884) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-3 "Executor #4 for ubuntu-test-VM-3" Id=57 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for master "Executor #5 for master" Id=43 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-1 "Executor #5 for ubuntu-test-VM-1" Id=69 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-2 "Executor #5 for ubuntu-test-VM-2" Id=80 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-3 "Executor #5 for ubuntu-test-VM-3" Id=58 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for master "Executor #6 for master" Id=44 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-1 "Executor #6 for ubuntu-test-VM-1" Id=70 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-2 "Executor #6 for ubuntu-test-VM-2" Id=81 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-3 "Executor #6 for ubuntu-test-VM-3" Id=59 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for master "Executor #7 for master" Id=45 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-1 "Executor #7 for ubuntu-test-VM-1" Id=71 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-2 "Executor #7 for ubuntu-test-VM-2" Id=82 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-3 "Executor #7 for ubuntu-test-VM-3" Id=60 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for master "Executor #8 for master" Id=46 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-1 "Executor #8 for ubuntu-test-VM-1" Id=72 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-2 "Executor #8 for ubuntu-test-VM-2" Id=83 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-3 "Executor #8 for ubuntu-test-VM-3" Id=61 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for master "Executor #9 for master" Id=47 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-1 "Executor #9 for ubuntu-test-VM-1" Id=73 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-2 "Executor #9 for ubuntu-test-VM-2" Id=84 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-3 "Executor #9 for ubuntu-test-VM-3" Id=62 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object.wait(Native Method) waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread2 "Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread2" Id=25 Group=main RUNNABLE at java.lang.String.intern(Native Method) at java.lang.Class.searchMethods(Class.java:2670) at java.lang.Class.getMethod0(Class.java:2694) at java.lang.Class.getMethod(Class.java:1622) at org.apache.commons.logging.LogFactory.directGetContextClassLoader(LogFactory.java:896) at org.apache.commons.logging.LogFactory$1.run(LogFactory.java:862) at java.security.AccessController.doPrivileged(Native Method) at org.apache.commons.logging.LogFactory.getContextClassLoaderInternal(LogFactory.java:859) at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:423) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:685) at org.apache.commons.beanutils.MethodUtils.getMatchingAccessibleMethod(MethodUtils.java:942) at org.apache.commons.beanutils.MethodUtils.invokeMethod(MethodUtils.java:274) at org.apache.commons.digester.SetNextRule.end(SetNextRule.java:220) at org.apache.commons.digester.Rule.end(Rule.java:257) at org.apache.commons.digester.Digester.endElement(Digester.java:1345) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:606) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1742) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2900) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:607) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:489) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:835) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:764) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1210) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:568) at org.apache.commons.digester.Digester.parse(Digester.java:1871) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:76) at hudson.scm.SubversionChangeLogParser.parse(SubversionChangeLogParser.java:43) at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:922) at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:896) at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:461) at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1581) at hudson.model.User.getProjects(User.java:449) at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21) at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:112) at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:532) at sun.reflect.GeneratedMethodAccessor242.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Property.writeValue(Property.java:232) at org.kohsuke.stapler.export.Property.writeValue(Property.java:137) at org.kohsuke.stapler.export.Property.writeTo(Property.java:114) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182) at org.kohsuke.stapler.export.Model.writeTo(Model.java:154) at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:264) at hudson.model.Api.doJson(Api.java:193) at sun.reflect.GeneratedMethodAccessor236.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:111) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)

at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:203) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770)

at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53)

at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)

at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@128d38f3 Handling GET /jenkins/threadDump : RequestHandlerThread7 "Handling GET /jenkins/threadDump : RequestHandlerThread7" Id=33 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@81e7b1d at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.FutureTask$Sync@81e7b1d at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1033) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1326) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:257) at java.util.concurrent.FutureTask.get(FutureTask.java:119) at hudson.remoting.LocalChannel$2.get(LocalChannel.java:81) at jenkins.model.Jenkins.getAllThreadDumps(Jenkins.java:2865) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.commons.jexl.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:258) at org.apache.commons.jexl.parser.ASTMethod.execute(ASTMethod.java:104) at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83) at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57) at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51) at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80) at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:74) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateRecurse(ExpressionSupport.java:61) at org.apache.commons.jelly.expression.ExpressionSupport.evaluateAsIterator(ExpressionSupport.java:94) at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:89) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98) at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91) at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99) at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119) at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105) at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63) at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53) at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)

at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32)

at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)

at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)

at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)

at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)

at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@567bd2f9 IM-Reconnector-Thread "IM-Reconnector-Thread" Id=21 Group=main WAITING on java.util.concurrent.Semaphore$NonfairSync@4c423981 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.Semaphore$NonfairSync@4c423981 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:160) at java.lang.Thread.run(Thread.java:722) javamelody jenkins "javamelody jenkins" Id=23 Group=main TIMED_WAITING on java.util.TaskQueue@5062da8 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@5062da8 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins cron thread "Jenkins cron thread" Id=16 Group=main TIMED_WAITING on java.util.TaskQueue@fcd6b44 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@fcd6b44 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=36 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.DatagramSocket.receive(DatagramSocket.java:786) locked java.net.DatagramPacket@4945a2e3 locked java.net.MulticastSocket@7ac90aa9 at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) JmDNS(FITBPWEB.local.).State.Timer "JmDNS(FITBPWEB.local.).State.Timer" Id=126 Group=main TIMED_WAITING on java.util.TaskQueue@187672e7 at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@187672e7 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) JmDNS(FITBPWEB.local.).Timer "JmDNS(FITBPWEB.local.).Timer" Id=125 Group=main TIMED_WAITING on java.util.TaskQueue@871a83d at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@871a83d at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) jrobin jenkins "jrobin jenkins" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@57f4be7d at java.lang.Object.wait(Native Method) waiting on java.util.TaskQueue@57f4be7d at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) LauncherControlThread[ControlPort=-1] "LauncherControlThread[ControlPort=-1]" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:722) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=34 Group=main RUNNABLE (in native) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) locked sun.nio.ch.Util$2@3298e7c locked java.util.Collections$UnmodifiableSet@43662b65 locked sun.nio.ch.WindowsSelectorImpl@5ee2fc3e at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@6ea937ab Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1 "Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1" Id=150 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3 "Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3" Id=158 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=149 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-11-thread-1 "pool-11-thread-1" Id=159 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-15-thread-1 "pool-15-thread-1" Id=118 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-3-thread-12 "pool-3-thread-12" Id=222 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.jenkinsci.plugins.vSphereCloudLauncher.launch(vSphereCloudLauncher.java:197) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@6de7a4d pool-3-thread-16 "pool-3-thread-16" Id=238 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@32a5f72a pool-4-thread-1 "pool-4-thread-1" Id=22 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-5-thread-1 "pool-5-thread-1" Id=235 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-1 "pool-7-thread-1" Id=225 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-2 "pool-7-thread-2" Id=226 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-3 "pool-7-thread-3" Id=241 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread14 "RequestHandlerThread14" Id=206 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread16 "RequestHandlerThread16" Id=208 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread20 "RequestHandlerThread20" Id=212 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread22 "RequestHandlerThread22" Id=214 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread23 "RequestHandlerThread23" Id=243 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread3 "RequestHandlerThread3" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) RequestHandlerThread6 "RequestHandlerThread6" Id=32 Group=main RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:427) locked java.util.zip.ZStreamRef@924b115 at java.util.zip.Deflater.deflate(Deflater.java:350) at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:251) at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146) locked java.util.zip.GZIPOutputStream@446b5a75 at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) locked java.io.OutputStreamWriter@325c9870 at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.PrintWriter.close(PrintWriter.java:339) locked java.io.OutputStreamWriter@325c9870

at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@51cbc1b5 SocketListener(FITBPWEB.local.) "SocketListener(FITBPWEB.local.)" Id=124 Group=main RUNNABLE (in native) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.DatagramSocket.receive(DatagramSocket.java:786) locked java.net.DatagramPacket@2f5cf3c8 locked java.net.MulticastSocket@7f88e7c2 at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) stderr copier for remote agent on serial-server-1 "stderr copier for remote agent on serial-server-1" Id=142 Group=main WAITING on java.lang.Object@3d0a76c2 at java.lang.Object.wait(Native Method) waiting on java.lang.Object@3d0a76c2 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) stderr copier for remote agent on ubuntu-test-VM-3 "stderr copier for remote agent on ubuntu-test-VM-3" Id=145 Group=main WAITING on java.lang.Object@ae4f2f4 at java.lang.Object.wait(Native Method) waiting on java.lang.Object@ae4f2f4 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) TCP slave agent listener port=0 "TCP slave agent listener port=0" Id=35 Group=main RUNNABLE (in native) at java.net.DualStackPlainSocketImpl.accept0(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) locked java.net.SocksSocketImpl@24db1c39 at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) Thread-10 "Thread-10" Id=127 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Thread-11 "Thread-11" Id=140 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Native Method) waiting on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-12 "Thread-12" Id=141 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Native Method) waiting on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-13 "Thread-13" Id=143 Group=main RUNNABLE at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-14 "Thread-14" Id=144 Group=main RUNNABLE at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-21 "Thread-21" Id=187 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) locked java.io.BufferedInputStream@20d1ed39 at com.sun.jndi.ldap.Connection.run(Connection.java:849) at java.lang.Thread.run(Thread.java:722) Thread-6 "Thread-6" Id=120 Group=main RUNNABLE at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475) at java.lang.Thread.run(Thread.java:722) Thread-7 "Thread-7" Id=121 Group=main RUNNABLE (in native) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at com.trilead.ssh2.crypto.cipher.CipherInputStream.fill_buffer(CipherInputStream.java:41) at com.trilead.ssh2.crypto.cipher.CipherInputStream.internal_read(CipherInputStream.java:52) at com.trilead.ssh2.crypto.cipher.CipherInputStream.getBlock(CipherInputStream.java:79) at com.trilead.ssh2.crypto.cipher.CipherInputStream.read(CipherInputStream.java:108) at com.trilead.ssh2.transport.TransportConnection.receiveMessage(TransportConnection.java:232) at com.trilead.ssh2.transport.TransportManager.receiveLoop(TransportManager.java:677) at com.trilead.ssh2.transport.TransportManager$1.run(TransportManager.java:475) at java.lang.Thread.run(Thread.java:722) ViewJob reload thread "ViewJob reload thread" Id=30 Group=main TIMED_WAITING on java.util.LinkedHashSet@5d1bfae4 at java.lang.Object.wait(Native Method) waiting on java.util.LinkedHashSet@5d1bfae4 at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:171) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:188) WinstoneHostConfigurationMgmt:default "WinstoneHostConfigurationMgmt:default" Id=11 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:722) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@22be5a1d at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@22be5a1d at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Java2D Disposer "Java2D Disposer" Id=9 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@579b8322 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@579b8322 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at sun.java2d.Disposer.run(Disposer.java:145) at java.lang.Thread.run(Thread.java:722) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7f0e37bc at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@7f0e37bc at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) D0-Debian-Slave-2 N/A offline D2-Debian-Slave-1 N/A offline D0-Debian-Slave-1 N/A offline D2-Debian-Slave-2 N/A offline serial-server-1 Channel reader thread: channel "Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:242) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) locked java.io.BufferedInputStream@1348b49 at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@6a21b2 at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@6a21b2 at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@6a21b2:channel "Ping thread for channel hudson.remoting.Channel@6a21b2:channel" Id=10 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

pool-1-thread-10 "pool-1-thread-10" Id=22 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

pool-1-thread-9 "pool-1-thread-9" Id=21 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722)

Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@1a970 Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@a49259 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@a49259 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@842f23 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@842f23 at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE D3-Debian-Slave-2 N/A offline ubuntu-test-VM-2 N/A offline ubuntu-test-VM-1 N/A offline D1-debian-FreshInstall-1 N/A offline D3-Debian-Slave-1 N/A offline ubuntu-test-VM-3 Channel reader thread: channel "Channel reader thread: channel" Id=8 Group=main RUNNABLE (in native) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:236) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) locked java.io.BufferedInputStream@12986ef at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@18d107f at java.lang.Object.wait(Native Method) waiting on hudson.remoting.Channel@18d107f at java.lang.Object.wait(Object.java:502) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@18d107f:channel "Ping thread for channel hudson.remoting.Channel@18d107f:channel" Id=9 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=11 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) pool-1-thread-3 "pool-1-thread-3" Id=14 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:392) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)

at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679) Number of locked synchronizers = 1 java.util.concurrent.ThreadPoolExecutor$Worker@5e305f pool-1-thread-4 "pool-1-thread-4" Id=15 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40 at sun.misc.Unsafe.park(Native Method) waiting on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:453) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:352) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@159cf26 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.ReferenceQueue$Lock@159cf26 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@12efed9 at java.lang.Object.wait(Native Method) waiting on java.lang.ref.Reference$Lock@12efed9 at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE

[elordahl@vt.edu (JIRA)]

Eric Lordahl commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I saw this for the first time today, and saw it twice. Three things have changed over the past week that could possibly affect this. I'm wondering how many people who have this problem use an NFS share, for I've seen a lot of file contention issues since the change to NFS. 1. (Few weeks ago)$JENKINS_HOME is now an NFS share 2. (Last night)Updated to 1.516 3. (Last night)Modified github plugin to use a, already-defined, ThreadFactory (see below): -private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Executors.newCachedThreadPool()); +private transient final SequentialExecutionQueue queue = new SequentialExecutionQueue(Computer.threadPoolForRemoting);

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@walterk82 indeed -Dorg.kohsuke.stapler.compression.CompressionFilter.disabled=true really only turns off the specialized error reporting, not the filter as a whole. For that you need to use -Dorg.kohsuke.stapler.jelly.DefaultScriptInvoker.compress=false from what I can tell.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Looking around for alternate implementations of GZIPOutputStream. JZlib looks most promising. Swapping this in for the native JRE implementation (zlib) should be trivial enough, but I need someone who is able to reproduce this bug and is willing to build Jenkins from sources (or give me a target Jenkins release version that I can build an alternate binary of) and compare the JZlib performance, so see if the hypothesis is valid that a bug in zlib is causing this issue.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Created http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.513.JENKINS-14362-jzlib/jenkins-war-1.513.JENKINS-14362-jzlib.war based on 1.513 if anyone wants to give it a whirl.

[yoda@schli.ch (JIRA)]

Marc Günther commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Guy Kisel: could you please wrap your stacktrace inside a {quote}, so this ticket remains readable?

[guy.kisel@gmail.com (JIRA)]

Guy Kisel edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Thread dump of this issue occuring on 1.516:

This message is automatically generated by JIRA.

[guy.kisel@gmail.com (JIRA)]

Guy Kisel edited a comment on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Thread dump of this issue occuring on 1.516:

Thread Dump AWT-Windows "AWT-Windows" Id=217 Group=main RUNNABLE (in native) at sun.awt.windows.WToolkit.eventLoop(Native Method) at sun.awt.windows.WToolkit.run(WToolkit.java:299) at java.lang.Thread.run(Thread.java:722) bot1-input "bot1-input" Id=167 Group=main RUNNABLE (in native

) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) at sun.security.ssl.InputRecord.read(InputRecord.java:480) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) - locked java.lang.Object@362d4a04 at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:884) at sun.security.ssl.AppInputStream.read(AppInputStream.java:102) - locked sun.security.ssl.AppInputStream@57566865 at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177) - locked java.io.InputStreamReader@69dfebae at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:154) at java.io.BufferedReader.readLine(BufferedReader.java:317) - locked java.io.InputStreamReader@69dfebae

at java.io.BufferedReader.readLine(BufferedReader.java:382) at org.pircbotx.InputThread.run(InputThread.java:83) bot1-output "bot1-output" Id=168 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6bea22cf

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.pircbotx.OutputThread.run(OutputThread.java:101) Channel reader thread: serial-server-1 "Channel reader thread: serial-server-1" Id=146 Group=main WAITING on java.lang.Object@61e52512 at java.lang.Object

.wait(Native Method) - waiting on java.lang.Object@61e52512 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) Channel reader thread: ubuntu-test-VM-3

"Channel reader thread: ubuntu-test-VM-3" Id=148 Group=main WAITING on java.lang.Object@1c5e0a0a at java.lang.Object

.wait(Native Method) - waiting on java.lang.Object@1c5e0a0a at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:144) at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2272) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2565) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2575) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) ConnectorThread:[http-8081]

"ConnectorThread:[http-8081]" Id=12 Group=main RUNNABLE (in native

) at java.net.DualStackPlainSocketImpl.waitForNewConnection(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:125) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) - locked java.net.SocksSocketImpl@773d4576

at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at winstone.HttpListener.run(HttpListener.java:139) at java.lang.Thread.run(Thread.java:722) DestroyJavaVM "DestroyJavaVM" Id=15 Group=main RUNNABLE Executor #0 for D0-Debian-Slave-1 "Executor #0 for D0-Debian-Slave-1" Id=86 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D0-Debian-Slave-2 "Executor #0 for D0-Debian-Slave-2" Id=91 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D1-debian-FreshInstall-1 "Executor #0 for D1-debian-FreshInstall-1" Id=116 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-1 "Executor #0 for D2-Debian-Slave-1" Id=111 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D2-Debian-Slave-2 "Executor #0 for D2-Debian-Slave-2" Id=106 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-1 "Executor #0 for D3-Debian-Slave-1" Id=96 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for D3-Debian-Slave-2 "Executor #0 for D3-Debian-Slave-2" Id=101 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for master "Executor #0 for master" Id=38 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for serial-server-1 "Executor #0 for serial-server-1" Id=48 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-1 "Executor #0 for ubuntu-test-VM-1" Id=64 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-2 "Executor #0 for ubuntu-test-VM-2" Id=75 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #0 for ubuntu-test-VM-3 "Executor #0 for ubuntu-test-VM-3" Id=53 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-1 "Executor #1 for D0-Debian-Slave-1" Id=87 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D0-Debian-Slave-2 "Executor #1 for D0-Debian-Slave-2" Id=92 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-1 "Executor #1 for D2-Debian-Slave-1" Id=112 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D2-Debian-Slave-2 "Executor #1 for D2-Debian-Slave-2" Id=107 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-1 "Executor #1 for D3-Debian-Slave-1" Id=97 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for D3-Debian-Slave-2 "Executor #1 for D3-Debian-Slave-2" Id=102 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for master "Executor #1 for master" Id=39 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for serial-server-1 "Executor #1 for serial-server-1" Id=49 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-1 "Executor #1 for ubuntu-test-VM-1" Id=65 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-2 "Executor #1 for ubuntu-test-VM-2" Id=76 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #1 for ubuntu-test-VM-3 "Executor #1 for ubuntu-test-VM-3" Id=54 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-1 "Executor #2 for D0-Debian-Slave-1" Id=88 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D0-Debian-Slave-2 "Executor #2 for D0-Debian-Slave-2" Id=93 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-1 "Executor #2 for D2-Debian-Slave-1" Id=113 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D2-Debian-Slave-2 "Executor #2 for D2-Debian-Slave-2" Id=108 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-1 "Executor #2 for D3-Debian-Slave-1" Id=98 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for D3-Debian-Slave-2 "Executor #2 for D3-Debian-Slave-2" Id=103 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for master "Executor #2 for master" Id=40 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for serial-server-1 "Executor #2 for serial-server-1" Id=50 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-1 "Executor #2 for ubuntu-test-VM-1" Id=66 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-2 "Executor #2 for ubuntu-test-VM-2" Id=77 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #2 for ubuntu-test-VM-3 "Executor #2 for ubuntu-test-VM-3" Id=55 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-1 "Executor #3 for D0-Debian-Slave-1" Id=89 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D0-Debian-Slave-2 "Executor #3 for D0-Debian-Slave-2" Id=94 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-1 "Executor #3 for D2-Debian-Slave-1" Id=114 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D2-Debian-Slave-2 "Executor #3 for D2-Debian-Slave-2" Id=109 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-1 "Executor #3 for D3-Debian-Slave-1" Id=99 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for D3-Debian-Slave-2 "Executor #3 for D3-Debian-Slave-2" Id=104 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for master "Executor #3 for master" Id=41 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for serial-server-1 "Executor #3 for serial-server-1" Id=51 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-1 "Executor #3 for ubuntu-test-VM-1" Id=67 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-2 "Executor #3 for ubuntu-test-VM-2" Id=78 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #3 for ubuntu-test-VM-3 "Executor #3 for ubuntu-test-VM-3" Id=56 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-1 "Executor #4 for D0-Debian-Slave-1" Id=90 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D0-Debian-Slave-2 "Executor #4 for D0-Debian-Slave-2" Id=95 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-1 "Executor #4 for D2-Debian-Slave-1" Id=115 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D2-Debian-Slave-2 "Executor #4 for D2-Debian-Slave-2" Id=110 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-1 "Executor #4 for D3-Debian-Slave-1" Id=100 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for D3-Debian-Slave-2 "Executor #4 for D3-Debian-Slave-2" Id=105 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for master "Executor #4 for master" Id=42 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-1 "Executor #4 for ubuntu-test-VM-1" Id=68 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-2 "Executor #4 for ubuntu-test-VM-2" Id=79 Group=main TIMED_WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at hudson.util.OneShotEvent.block(OneShotEvent.java:86) at hudson.model.Queue.pop(Queue.java:884) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #4 for ubuntu-test-VM-3 "Executor #4 for ubuntu-test-VM-3" Id=57 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for master "Executor #5 for master" Id=43 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-1 "Executor #5 for ubuntu-test-VM-1" Id=69 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-2 "Executor #5 for ubuntu-test-VM-2" Id=80 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #5 for ubuntu-test-VM-3 "Executor #5 for ubuntu-test-VM-3" Id=58 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for master "Executor #6 for master" Id=44 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-1 "Executor #6 for ubuntu-test-VM-1" Id=70 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-2 "Executor #6 for ubuntu-test-VM-2" Id=81 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #6 for ubuntu-test-VM-3 "Executor #6 for ubuntu-test-VM-3" Id=59 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for master "Executor #7 for master" Id=45 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-1 "Executor #7 for ubuntu-test-VM-1" Id=71 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-2 "Executor #7 for ubuntu-test-VM-2" Id=82 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #7 for ubuntu-test-VM-3 "Executor #7 for ubuntu-test-VM-3" Id=60 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for master "Executor #8 for master" Id=46 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-1 "Executor #8 for ubuntu-test-VM-1" Id=72 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-2 "Executor #8 for ubuntu-test-VM-2" Id=83 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #8 for ubuntu-test-VM-3 "Executor #8 for ubuntu-test-VM-3" Id=61 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for master "Executor #9 for master" Id=47 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-1 "Executor #9 for ubuntu-test-VM-1" Id=73 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-2 "Executor #9 for ubuntu-test-VM-2" Id=84 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238

at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Executor #9 for ubuntu-test-VM-3 "Executor #9 for ubuntu-test-VM-3" Id=62 Group=main WAITING on hudson.model.Queue@156b238 at java.lang.Object

.wait(Native Method) - waiting on hudson.model.Queue@156b238 at java.lang.Object.wait(Object.java:503) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) at hudson.model.Queue.pop(Queue.java:882) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2] "Handling GET /jenkins//job/Hardware_Stability_Tests_Sequoia/api/json : RequestHandlerThread[#2]" Id=25 Group=main RUNNABLE

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@128d38f3 Handling GET /jenkins/threadDump : RequestHandlerThread[#7] "Handling GET /jenkins/threadDump : RequestHandlerThread[#7]" Id=33 Group=main TIMED_WAITING on java.util.concurrent.FutureTask$Sync@81e7b1d at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.FutureTask$Sync@81e7b1d

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@567bd2f9

IM-Reconnector-Thread "IM-Reconnector-Thread" Id=21 Group=main WAITING on java.util.concurrent.Semaphore$NonfairSync@4c423981

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.Semaphore$NonfairSync@4c423981

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) at hudson.plugins.im.IMConnectionProvider$ConnectorRunnable.run(IMConnectionProvider.java:160) at java.lang.Thread.run(Thread.java:722) javamelody jenkins "javamelody jenkins" Id=23 Group=main TIMED_WAITING on java.util.TaskQueue@5062da8 at java.lang.Object

.wait(Native Method) - waiting on java.util.TaskQueue@5062da8

at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins cron thread "Jenkins cron thread" Id=16 Group=main TIMED_WAITING on java.util.TaskQueue@fcd6b44 at java.lang.Object

.wait(Native Method) - waiting on java.util.TaskQueue@fcd6b44

at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) Jenkins UDP 33848 monitoring thread "Jenkins UDP 33848 monitoring thread" Id=36 Group=main RUNNABLE (in native

) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) - locked java.net.TwoStacksPlainDatagramSocketImpl@62ce7b37 at java.net.DatagramSocket.receive(DatagramSocket.java:786) - locked java.net.DatagramPacket@4945a2e3 - locked java.net.MulticastSocket@7ac90aa9

at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) JmDNS(FITBPWEB.local.).State.Timer "JmDNS(FITBPWEB.local.).State.Timer" Id=126 Group=main TIMED_WAITING on java.util.TaskQueue@187672e7 at java.lang.Object

.wait(Native Method) - waiting on java.util.TaskQueue@187672e7

at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) JmDNS(FITBPWEB.local.).Timer "JmDNS(FITBPWEB.local.).Timer" Id=125 Group=main TIMED_WAITING on java.util.TaskQueue@871a83d at java.lang.Object

.wait(Native Method) - waiting on java.util.TaskQueue@871a83d

at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) jrobin jenkins "jrobin jenkins" Id=27 Group=main TIMED_WAITING on java.util.TaskQueue@57f4be7d at java.lang.Object

.wait(Native Method) - waiting on java.util.TaskQueue@57f4be7d

at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) LauncherControlThread[ControlPort=-1] "LauncherControlThread[ControlPort=-1]" Id=13 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:722) NioSocketAcceptor-1 "NioSocketAcceptor-1" Id=34 Group=main RUNNABLE (in native

) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295) at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277) at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@3298e7c - locked java.util.Collections$UnmodifiableSet@43662b65 - locked sun.nio.ch.WindowsSelectorImpl@5ee2fc3e

at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6ea937ab

Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1 "Ping thread for channel hudson.remoting.Channel@246e3870:serial-server-1" Id=150 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3 "Ping thread for channel hudson.remoting.Channel@660da176:ubuntu-test-VM-3" Id=158 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) pool-1-svnkit-thread-1 "pool-1-svnkit-thread-1" Id=149 Group=main WAITING on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@3a898c5c

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-11-thread-1 "pool-11-thread-1" Id=159 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58937b62

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-15-thread-1 "pool-15-thread-1" Id=118 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7527b2f7

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1079) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-3-thread-12 "pool-3-thread-12" Id=222 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at org.jenkinsci.plugins.vSphereCloudLauncher.launch(vSphereCloudLauncher.java:197) at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:223) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@6de7a4d

pool-3-thread-16 "pool-3-thread-16" Id=238 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.LocalChannel$1.call(LocalChannel.java:52) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@32a5f72a

pool-4-thread-1 "pool-4-thread-1" Id=22 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@589f732b

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-5-thread-1 "pool-5-thread-1" Id=235 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@341ce778

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-1 "pool-7-thread-1" Id=225 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-2 "pool-7-thread-2" Id=226 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) pool-7-thread-3 "pool-7-thread-3" Id=241 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ea58288

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#14] "RequestHandlerThread[#14]" Id=206 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#16] "RequestHandlerThread[#16]" Id=208 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#20] "RequestHandlerThread[#20]" Id=212 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#22] "RequestHandlerThread[#22]" Id=214 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#23] "RequestHandlerThread[#23]" Id=243 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#3] "RequestHandlerThread[#3]" Id=26 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7becaa5f

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread

.java:722) RequestHandlerThread[#6] "RequestHandlerThread[#6]" Id=32 Group=main RUNNABLE at java.util.zip.Deflater.deflateBytes(Native Method) at java.util.zip.Deflater.deflate(Deflater.java:427) - locked java.util.zip.ZStreamRef@924b115 at java.util.zip.Deflater.deflate(Deflater.java:350) at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:251) at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146) - locked java.util.zip.GZIPOutputStream@446b5a75 at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316) at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149) - locked java.io.OutputStreamWriter@325c9870 at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233) at java.io.PrintWriter.close(PrintWriter.java:339) - locked java.io.OutputStreamWriter@325c9870

at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:78) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Number

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@51cbc1b5

SocketListener(FITBPWEB.local.) "SocketListener(FITBPWEB.local.)" Id=124 Group=main RUNNABLE (in native

) at java.net.TwoStacksPlainDatagramSocketImpl.receive0(Native Method) - locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.TwoStacksPlainDatagramSocketImpl.receive(TwoStacksPlainDatagramSocketImpl.java:90) - locked java.net.TwoStacksPlainDatagramSocketImpl@5ddc411f at java.net.DatagramSocket.receive(DatagramSocket.java:786) - locked java.net.DatagramPacket@2f5cf3c8 - locked java.net.MulticastSocket@7f88e7c2

at javax.jmdns.impl.SocketListener.run(SocketListener.java:41) stderr copier for remote agent on serial-server-1 "stderr copier for remote agent on serial-server-1" Id=142 Group=main WAITING on java.lang.Object@3d0a76c2 at java.lang.Object

.wait(Native Method) - waiting on java.lang.Object@3d0a76c2 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) stderr copier for remote agent on ubuntu-test-VM-3

"stderr copier for remote agent on ubuntu-test-VM-3" Id=145 Group=main WAITING on java.lang.Object@ae4f2f4 at java.lang.Object

.wait(Native Method) - waiting on java.lang.Object@ae4f2f4 at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:213) at com.trilead.ssh2.StreamGobbler.read(StreamGobbler.java:171) at hudson.util.StreamCopyThread.run(StreamCopyThread.java:60) TCP slave agent listener port=0

"TCP slave agent listener port=0" Id=35 Group=main RUNNABLE (in native

) at java.net.DualStackPlainSocketImpl.accept0(Native Method) at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183) - locked java.net.SocksSocketImpl@24db1c39

at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) Thread-10 "Thread-10" Id=127 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43f53914

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Thread-11 "Thread-11" Id=140 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object

.wait(Native Method) - waiting on com.trilead.ssh2.channel.Channel@79fad9dc

at java.lang.Object.wait(Object.java:503) at com.trilead.ssh2.channel.FifoBuffer.read(FifoBuffer.java:210) at com.trilead.ssh2.channel.ChannelManager.getChannelData(ChannelManager.java:961) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:58) at com.trilead.ssh2.channel.ChannelInputStream.read(ChannelInputStream.java:70) at com.trilead.ssh2.StreamGobbler$GobblerThread.run(StreamGobbler.java:48) Thread-12 "Thread-12" Id=141 Group=main WAITING on com.trilead.ssh2.channel.Channel@79fad9dc at java.lang.Object

.wait(Native Method) - waiting on com.trilead.ssh2.channel.Channel@79fad9dc

) at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:150) at java.net.SocketInputStream.read(SocketInputStream.java:121) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read1(BufferedInputStream.java:275) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - locked java.io.BufferedInputStream@20d1ed39

.wait(Native Method) - waiting on java.util.LinkedHashSet@5d1bfae4

at hudson.model.ViewJob$ReloadThread.getNext(ViewJob.java:171) at hudson.model.ViewJob$ReloadThread.run(ViewJob.java:188) WinstoneHostConfigurationMgmt:default "WinstoneHostConfigurationMgmt:default" Id=11 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:722) Attach Listener "Attach Listener" Id=4 Group=system RUNNABLE Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@22be5a1d at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@22be5a1d

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Java2D Disposer "Java2D Disposer" Id=9 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@579b8322 at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@579b8322

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at sun.java2d.Disposer.run(Disposer.java:145) at java.lang.Thread.run(Thread.java:722) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@7f0e37bc at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@7f0e37bc

at java.lang.Object.wait(Object.java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) D0-Debian-Slave-2 N/A offline D2-Debian-Slave-1 N/A offline D0-Debian-Slave-1 N/A offline D2-Debian-Slave-2 N/A offline serial-server-1 Channel reader thread: channel "Channel reader thread: channel" Id=9 Group=main RUNNABLE (in native

) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:242) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) - locked java.io.BufferedInputStream@1348b49

at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2559) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2569) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1315) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@6a21b2 at java.lang.Object

.wait(Native Method) - waiting on hudson.remoting.Channel@6a21b2

at java.lang.Object.wait(Object.java:503) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@6a21b2:channel "Ping thread for channel hudson.remoting.Channel@6a21b2:channel" Id=10 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=12 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1d66f88

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) pool-1-thread-10 "pool-1-thread-10" Id=22 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17e21b3

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@1a970

Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@a49259 at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@a49259

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@842f23 at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@842f23

) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:236) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) - locked java.io.BufferedInputStream@12986ef

at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2265) at java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:2558) at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2568) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368) at hudson.remoting.Command.readFrom(Command.java:92) at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48) main "main" Id=1 Group=main WAITING on hudson.remoting.Channel@18d107f at java.lang.Object

.wait(Native Method) - waiting on hudson.remoting.Channel@18d107f

at java.lang.Object.wait(Object.java:502) at hudson.remoting.Channel.join(Channel.java:800) at hudson.remoting.Launcher.main(Launcher.java:484) at hudson.remoting.Launcher.runWithStdinStdout(Launcher.java:430) at hudson.remoting.Launcher.run(Launcher.java:221) at hudson.remoting.Launcher.main(Launcher.java:180) Ping thread for channel hudson.remoting.Channel@18d107f:channel "Ping thread for channel hudson.remoting.Channel@18d107f:channel" Id=9 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:86) Pipe writer thread: channel "Pipe writer thread: channel" Id=11 Group=main WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6d156

at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) pool-1-thread-3 "pool-1-thread-3" Id=14 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:392) at hudson.Functions.getThreadInfos(Functions.java:1061) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:96) at hudson.util.RemotingDiagnostics$GetThreadDump.call(RemotingDiagnostics.java:92) at hudson.remoting.UserRequest.perform(UserRequest.java:118) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:326) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Number

of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@5e305f

pool-1-thread-4 "pool-1-thread-4" Id=15 Group=main TIMED_WAITING on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40

at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@7a7e40

at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:453) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:352) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:679) Finalizer "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@159cf26 at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@159cf26

at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:133) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:149) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) Reference Handler "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@12efed9 at java.lang.Object

.wait(Native Method) - waiting on java.lang.ref.Reference$Lock@12efed9

at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) Signal Dispatcher "Signal Dispatcher" Id=4 Group=system RUNNABLE

[guy.kisel@gmail.com (JIRA)]

Guy Kisel commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Fixed. Apologies.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Thanks @guykisel; also in the future consider just attaching files.

[yoda@schli.ch (JIRA)]

Marc Günther edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Guy Kisel: could you please wrap your stacktrace inside a {quote}, so this ticket remains readable? EDIT: Thanks a lot

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Filed https://github.com/stapler/stapler/pull/22 and https://github.com/jenkinsci/jenkins/pull/804.

[jglick@cloudbees.com (JIRA)]

Jesse Glick assigned JENKINS-14362 to Jesse Glick

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Change By: Jesse Glick (11/Jun/13 12:02 PM) Assignee: Jesse Glick

[jglick@cloudbees.com (JIRA)]

Jesse Glick started work on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Change By: Jesse Glick (11/Jun/13 12:02 PM)

Status: Open In Progress

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Change By: Jesse Glick (11/Jun/13 12:02 PM)

Labels: lts-candidate performance

[jglick@cloudbees.com (JIRA)]

Jesse Glick updated JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Upgrading priority since this has been reported to necessitate Jenkins restarts to avoid excess load after a few days. Change By: Jesse Glick (11/Jun/13 12:06 PM) Priority: Major Critical

[elordahl@vt.edu (JIRA)]

Eric Lordahl commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I've been using the "Monitoring" plugin to terminate the corresponding thread, instead of restarting Jenkins. The thread will be named something like "requestHandlerThreadxyz" and will most likely have the most cpu/user time by a long shot.

[elordahl@vt.edu (JIRA)]

Eric Lordahl commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@jglick - Can you build a 1.516 WAR with the fix? If so,I can test that Friday morning. I don't know what specifically is triggering the issue in our instance, but it repeatedly occurs during a specific segment of our build cycle.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@elordahl: http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.516.JENKINS-14362-jzlib/jenkins-war-1.516.JENKINS-14362-jzlib.war

[elordahl@vt.edu (JIRA)]

Eric Lordahl edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

I've been using the "Monitoring" plugin to terminate the corresponding thread, instead of restarting Jenkins. The thread will be named something like "requestHandlerThread[#xyz]" and will most likely have the most cpu/user time by a long shot.

[elordahl@vt.edu (JIRA)]

Eric Lordahl commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

With the jenkins-war-1.516.JENKINS-14362-jzlib.war running, we did not see the hanging threads or CPUs @ 100%. Good sign. The next time I'll be able to test again will be Tuesday morning. I'll also gladly install a 1.518 build w/ this fix as 1.518 has the css issue resolved @jglick - Let me know if you build one.

[joncrookesites@gmail.com (JIRA)]

Jonathan Crooke commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

jenkins-war-1.516.JENKINS-14362-jzlib.war doesn't resolve this issue for me. CPU usage still peaks and Jenkins becomes locked out. Here's my stack trace: 2013-06-17 12:37:37 Full thread dump Java HotSpot(TM) 64-Bit Server VM (20.45-b01-451 mixed mode): "Attach Listener" daemon prio=9 tid=7ff14f801000 nid=0x11c9b5000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "RequestHandlerThread[#12]" daemon prio=5 tid=7ff14e806000 nid=0x11f57b000 waiting on condition [11f57a000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "Handling GET /adjuncts/cb752a56/lib/layout/menu_right_arrow.png : RequestHandlerThread[#11]" daemon prio=5 tid=7ff14e9ed000 nid=0x11f926000 runnable [11f922000] java.lang.Thread.State: RUNNABLE at java.io.File.<init>(File.java:318) at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:996) at sun.misc.URLClassPath$FileLoader.findResource(URLClassPath.java:966) at sun.misc.URLClassPath.findResource(URLClassPath.java:146) at java.net.URLClassLoader$2.run(URLClassLoader.java:385) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findResource(URLClassLoader.java:382) at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at jenkins.ClassLoaderReflectionToolkit.findResource(ClassLoaderReflectionToolkit.java:52) at hudson.PluginManager$UberClassLoader.findResource(PluginManager.java:972) at java.lang.ClassLoader.getResource(ClassLoader.java:1002) at org.kohsuke.stapler.framework.adjunct.AdjunctManager.allowResourceToBeServed(AdjunctManager.java:198) at org.kohsuke.stapler.framework.adjunct.AdjunctManager.doDynamic(AdjunctManager.java:165) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:288) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:151) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:90) at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:363) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:241) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:677) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:770) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:583) at org.kohsuke.stapler.Stapler.service(Stapler.java:214) at javax.servlet.http.HttpServlet.service(HttpServlet.java:45) at winstone.ServletConfiguration.execute(ServletConfiguration.java:248) at winstone.RequestDispatcher.forward(RequestDispatcher.java:333) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:36) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:32) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:206) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:179) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:86) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:84) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:64) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81) at winstone.FilterConfiguration.execute(FilterConfiguration.java:194) at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366) at winstone.RequestDispatcher.forward(RequestDispatcher.java:331) at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:227) at winstone.RequestHandlerThread.run(RequestHandlerThread.java:150) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#10]" daemon prio=5 tid=7ff14ff93000 nid=0x11c2f1000 waiting on condition [11c2f0000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#9]" daemon prio=5 tid=7ff14ff92800 nid=0x11c8b2000 waiting on condition [11c8b1000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "pool-10-thread-1" prio=5 tid=7ff14ff37800 nid=0x11f823000 waiting on condition [11f822000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77e07af48> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "JmDNS(Cornelius-Builder-Monkey-local.local.).State.Timer" prio=5 tid=7ff14cc94000 nid=0x12037d000 in Object.wait() [12037c000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77e5be5c0> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked <77e5be5c0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "JmDNS(Cornelius-Builder-Monkey-local.local.).Timer" daemon prio=5 tid=7ff14cada800 nid=0x11cabd000 runnable [11cabc000] java.lang.Thread.State: RUNNABLE at javax.jmdns.impl.DNSCache.allValues(DNSCache.java:349) - locked <77e5be668> (a javax.jmdns.impl.DNSCache) at javax.jmdns.impl.JmDNSImpl.cleanCache(JmDNSImpl.java:1771) at javax.jmdns.impl.tasks.RecordReaper.run(RecordReaper.java:58) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) "pool-16-thread-1" daemon prio=5 tid=7ff14ce6e800 nid=0x120177000 waiting on condition [120176000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77e686eb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) at java.util.concurrent.DelayQueue.take(DelayQueue.java:160) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "Executor #1 for master" prio=5 tid=7ff14ce6d800 nid=0x120074000 in Object.wait() [120073000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77da4b940> (a hudson.model.Queue) at java.lang.Object.wait(Object.java:485) at hudson.util.OneShotEvent.block(OneShotEvent.java:72) - locked <77da4b940> (a hudson.model.Queue) at hudson.model.Queue.pop(Queue.java:882) - locked <77da4b940> (a hudson.model.Queue) at hudson.model.Executor.grabJob(Executor.java:289) at hudson.model.Executor.run(Executor.java:210) - locked <77da4b940> (a hudson.model.Queue) "Jenkins UDP 33848 monitoring thread" prio=5 tid=7ff14ce69800 nid=0x11fd6b000 runnable [11fd6a000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <77e466810> (a java.net.PlainDatagramSocketImpl) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) - locked <77e466810> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:725) - locked <77e4ee470> (a java.net.DatagramPacket) - locked <77e6128e8> (a java.net.MulticastSocket) at hudson.UDPBroadcastThread.run(UDPBroadcastThread.java:82) "TCP slave agent listener port=0" prio=5 tid=7ff14ca42800 nid=0x11f478000 runnable [11f477000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <77e64d808> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at hudson.TcpSlaveAgentListener.run(TcpSlaveAgentListener.java:91) "NioSocketAcceptor-1" prio=5 tid=7ff14d0d8800 nid=0x11f2db000 runnable [11f2da000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <77e50f7d8> (a sun.nio.ch.Util$2) - locked <77e50f7c0> (a java.util.Collections$UnmodifiableSet) - locked <77e5f2d88> (a sun.nio.ch.KQueueSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) at org.apache.mina.transport.socket.nio.NioSocketAcceptor.select(NioSocketAcceptor.java:238) at org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:432) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:680) "RequestHandlerThread[#5]" daemon prio=5 tid=7ff14d028000 nid=0x11f720000 waiting on condition [11f71f000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <77d8c4f40> (a java.util.concurrent.SynchronousQueue$TransferStack) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:422) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:323) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:857) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:917) at java.lang.Thread.run(Thread.java:680) "jrobin " daemon prio=5 tid=7ff14d889800 nid=0x119f93000 in Object.wait() [119f92000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77dcfd2b0> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:485) at java.util.TimerThread.mainLoop(Timer.java:483) - locked <77dcfd2b0> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "DestroyJavaVM" prio=5 tid=7ff14bdb8800 nid=0x1103ce000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "LauncherControlThread[ControlPort=-1]" prio=5 tid=7ff14bdb7800 nid=0x11c6d3000 waiting on condition [11c6d2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.Launcher.run(Launcher.java:265) at java.lang.Thread.run(Thread.java:680) "ConnectorThread:[http-8080]" daemon prio=5 tid=7ff14cea9000 nid=0x11c5d0000 runnable [11c5cf000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) - locked <77d8b0530> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at winstone.HttpListener.run(HttpListener.java:139) at java.lang.Thread.run(Thread.java:680) "WinstoneHostConfigurationMgmt:default" daemon prio=5 tid=7ff14cf30000 nid=0x11c4cd000 waiting on condition [11c4cc000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at winstone.HostConfiguration.run(HostConfiguration.java:174) at java.lang.Thread.run(Thread.java:680) "Java2D Disposer" daemon prio=10 tid=7ff14bd19800 nid=0x11ac46000 in Object.wait() [11ac45000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77db5c678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <77db5c678> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at sun.java2d.Disposer.run(Disposer.java:127) at java.lang.Thread.run(Thread.java:680) "AWT-AppKit" daemon prio=5 tid=7ff14ce3d000 nid=0x7fff74186180 runnable [00000000] java.lang.Thread.State: RUNNABLE "Low Memory Detector" daemon prio=5 tid=7ff14c87f000 nid=0x119b64000 runnable [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=9 tid=7ff14c87e000 nid=0x119a61000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=9 tid=7ff14c87d800 nid=0x11995e000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=9 tid=7ff14c87c800 nid=0x11985b000 runnable [00000000] java.lang.Thread.State: RUNNABLE "Surrogate Locker Thread (Concurrent GC)" daemon prio=5 tid=7ff14c87b800 nid=0x119758000 waiting on condition [00000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=8 tid=7ff14b840800 nid=0x1194ca000 in Object.wait() [1194c9000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77d875d58> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <77d875d58> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:171) "Reference Handler" daemon prio=10 tid=7ff14b83f800 nid=0x1193c7000 in Object.wait() [1193c6000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on <77d86b5a8> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <77d86b5a8> (a java.lang.ref.Reference$Lock) "VM Thread" prio=9 tid=7ff14b83d000 nid=0x1192c4000 runnable "Gang worker#0 (Parallel GC Threads)" prio=9 tid=7ff14c801800 nid=0x1137d2000 runnable "Gang worker#1 (Parallel GC Threads)" prio=9 tid=7ff14c802000 nid=0x1138d5000 runnable "Concurrent Mark-Sweep GC Thread" prio=9 tid=7ff14c84d000 nid=0x118f3e000 runnable "VM Periodic Task Thread" prio=10 tid=7ff14c888800 nid=0x119c67000 waiting on condition "Exception Catcher Thread" prio=10 tid=7ff14c801000 nid=0x1105f9000 runnable JNI global references: 1556

[joncrookesites@gmail.com (JIRA)]

Jonathan Crooke edited a comment on JENKINS-14362

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@joncrooke your issue looks unrelated and should be filed separately. Use your browser’s debug view to see if /adjuncts/cb752a56/lib/layout/menu_right_arrow.png is being served with an appropriate cache header; ought to be loaded once as a 200 and then get 304 NOT MODIFIED thereafter.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@elordahl: http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.518.JENKINS-14362-jzlib/jenkins-war-1.518.JENKINS-14362-jzlib.war

[dogfood@java.net (JIRA)]

dogfood commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Integrated in jenkins_main_trunk #2629 [FIXED JENKINS-17713 JENKINS-14362] (Revision bda95c67a6780ff0e6350f80d6f69cf0f61e737c) Result = SUCCESS kohsuke : bda95c67a6780ff0e6350f80d6f69cf0f61e737c Files : core/pom.xml changelog.html

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html core/pom.xml http://jenkins-ci.org/commit/jenkins/bda95c67a6780ff0e6350f80d6f69cf0f61e737c Log: [FIXED JENKINS-17713 JENKINS-14362] Integrated the new version of Stapler that fixes them. Compare: https://github.com/jenkinsci/jenkins/compare/59d4c6a4c25a...bda95c67a678

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Jesse Glick Path: cli/pom.xml core/pom.xml core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/model/UsageStatistics.java core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java core/src/main/java/hudson/util/CompressedFile.java maven-plugin/pom.xml plugins/pom.xml pom.xml test/pom.xml ui-samples-plugin/pom.xml war/pom.xml http://jenkins-ci.org/commit/jenkins/3701a327a75d35e5d0e5795341cc55bb123a2007 Log: JENKINS-14362 Switch to jzlib for GZIP streaming.

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Jesse Glick Path:

changelog.html cli/pom.xml cli/src/main/java/hudson/cli/CLI.java core/pom.xml core/src/main/java/hudson/AboutJenkins.java core/src/main/java/hudson/ClassicPluginStrategy.java core/src/main/java/hudson/ExtensionFinder.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Functions.java core/src/main/java/hudson/PluginManager.java core/src/main/java/hudson/Util.java core/src/main/java/hudson/WebAppMain.java core/src/main/java/hudson/cli/ClientAuthenticationCache.java core/src/main/java/hudson/cli/CommandDuringBuild.java core/src/main/java/hudson/cli/SetBuildParameterCommand.java core/src/main/java/hudson/init/impl/GroovyInitScript.java core/src/main/java/hudson/logging/LogRecorder.java core/src/main/java/hudson/markup/MyspacePolicy.java core/src/main/java/hudson/matrix/Combination.java core/src/main/java/hudson/matrix/DefaultMatrixExecutionStrategyImpl.java core/src/main/java/hudson/matrix/MatrixBuild.java core/src/main/java/hudson/matrix/MatrixConfiguration.java core/src/main/java/hudson/matrix/MatrixProject.java core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/AbstractItem.java core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Cause.java core/src/main/java/hudson/model/CauseAction.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/ComputerSet.java core/src/main/java/hudson/model/Descriptor.java core/src/main/java/hudson/model/DisplayNameListener.java core/src/main/java/hudson/model/Executor.java core/src/main/java/hudson/model/FileParameterDefinition.java core/src/main/java/hudson/model/FileParameterValue.java core/src/main/java/hudson/model/Fingerprint.java core/src/main/java/hudson/model/ItemGroup.java core/src/main/java/hudson/model/JDK.java core/src/main/java/hudson/model/Job.java core/src/main/java/hudson/model/ListView.java core/src/main/java/hudson/model/Node.java core/src/main/java/hudson/model/ParametersAction.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java core/src/main/java/hudson/model/Queue.java core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/model/RunAction.java core/src/main/java/hudson/model/RunMap.java core/src/main/java/hudson/model/RunParameterDefinition.java core/src/main/java/hudson/model/RunParameterValue.java core/src/main/java/hudson/model/StreamBuildListener.java core/src/main/java/hudson/model/UpdateCenter.java core/src/main/java/hudson/model/View.java core/src/main/java/hudson/model/queue/CauseOfBlockage.java core/src/main/java/hudson/model/queue/WorkUnit.java core/src/main/java/hudson/os/PosixAPI.java core/src/main/java/hudson/os/PosixException.java core/src/main/java/hudson/scheduler/CronTab.java core/src/main/java/hudson/search/Search.java core/src/main/java/hudson/search/SearchItem.java core/src/main/java/hudson/security/BasicAuthenticationFilter.java core/src/main/java/hudson/security/FederatedLoginService.java core/src/main/java/hudson/security/HudsonPrivateSecurityRealm.java core/src/main/java/hudson/security/Permission.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/hudson/tasks/Fingerprinter.java core/src/main/java/hudson/tasks/Maven.java core/src/main/java/hudson/tasks/UserAvatarResolver.java core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/java/hudson/tasks/junit/ClassResult.java core/src/main/java/hudson/tasks/junit/JUnitParser.java core/src/main/java/hudson/tasks/junit/PackageResult.java core/src/main/java/hudson/tasks/junit/TestNameTransformer.java core/src/main/java/hudson/tasks/test/TestResult.java core/src/main/java/hudson/tools/CommandInstaller.java core/src/main/java/hudson/tools/ZipExtractionInstaller.java core/src/main/java/hudson/util/CopyOnWriteMap.java core/src/main/java/hudson/util/FormFieldValidator.java core/src/main/java/hudson/util/FormValidation.java core/src/main/java/hudson/util/IOUtils.java core/src/main/java/hudson/util/PersistedList.java core/src/main/java/hudson/util/RunList.java core/src/main/java/hudson/util/XStream2.java core/src/main/java/hudson/util/jna/GNUCLibrary.java core/src/main/java/hudson/widgets/Widget.java core/src/main/java/jenkins/model/Jenkins.java core/src/main/java/jenkins/model/PeepholePermalink.java core/src/main/java/jenkins/model/RunAction2.java core/src/main/java/jenkins/model/lazy/AbstractLazyLoadRunMap.java core/src/main/java/jenkins/model/lazy/Boundary.java core/src/main/java/jenkins/model/lazy/BuildReferenceMapAdapter.java core/src/main/java/jenkins/mvn/GlobalMavenConfig.java core/src/main/java/jenkins/security/ConfidentialStore.java core/src/main/java/jenkins/security/RekeySecretAdminMonitor.java core/src/main/java/jenkins/util/JSONSignatureValidator.java core/src/main/java/jenkins/util/ProgressiveRendering.java core/src/main/java/jenkins/widgets/BuildQueueWidget.java core/src/main/java/jenkins/widgets/ExecutorsWidget.java core/src/main/resources/dsld/IntelliJ.gdsl core/src/main/resources/hudson/AboutJenkins/index.jelly core/src/main/resources/hudson/AboutJenkins/index.properties core/src/main/resources/hudson/AboutJenkins/index_fr.properties core/src/main/resources/hudson/Messages.properties core/src/main/resources/hudson/Messages_de.properties core/src/main/resources/hudson/Messages_fr.properties core/src/main/resources/hudson/PluginManager/advanced.jelly core/src/main/resources/hudson/PluginManager/installed.jelly core/src/main/resources/hudson/cli/Messages.properties core/src/main/resources/hudson/logging/LogRecorder/configure.jelly core/src/main/resources/hudson/logging/LogRecorder/index.jelly core/src/main/resources/hudson/matrix/MatrixBuild/ajaxMatrix_fr.properties core/src/main/resources/hudson/matrix/MatrixProject/ajaxMatrix_fr.properties core/src/main/resources/hudson/matrix/MatrixProject/index.jelly core/src/main/resources/hudson/matrix/Messages.properties core/src/main/resources/hudson/model/Cause/UpstreamCause/description_pl.properties core/src/main/resources/hudson/model/Cause/UserCause/description_pl.properties core/src/main/resources/hudson/model/Cause/UserIdCause/description_pl.properties core/src/main/resources/hudson/model/FileParameterDefinition/config_fr.properties core/src/main/resources/hudson/model/FileParameterValue/value_fr.properties core/src/main/resources/hudson/model/FreeStyleProject/newJobDetail_fr.properties core/src/main/resources/hudson/model/Job/_api.jelly core/src/main/resources/hudson/model/Job/buildTimeTrend_fr.properties core/src/main/resources/hudson/model/Job/configure_fr.properties core/src/main/resources/hudson/model/Job/index.jelly core/src/main/resources/hudson/model/Job/index_fr.properties core/src/main/resources/hudson/model/Messages.properties core/src/main/resources/hudson/model/Messages_de.properties core/src/main/resources/hudson/model/Messages_fr.properties core/src/main/resources/hudson/model/ParametersAction/index_fr.properties core/src/main/resources/hudson/model/ParametersDefinitionProperty/index_fr.properties core/src/main/resources/hudson/model/PasswordParameterDefinition/config_fr.properties core/src/main/resources/hudson/model/RunParameterDefinition/config.jelly core/src/main/resources/hudson/model/RunParameterDefinition/index.jelly core/src/main/resources/hudson/model/View/AsynchPeople/index.jelly core/src/main/resources/hudson/model/View/AsynchPeople/index.properties core/src/main/resources/hudson/model/View/sidepanel.jelly core/src/main/resources/hudson/model/labels/LabelAtom/configure_fr.properties core/src/main/resources/hudson/slaves/JNLPLauncher/main.jelly core/src/main/resources/hudson/slaves/Messages_fr.properties core/src/main/resources/hudson/slaves/OfflineCause/LaunchFailed/cause_fr.properties core/src/main/resources/hudson/slaves/SlaveComputer/log.jelly core/src/main/resources/hudson/slaves/SlaveComputer/statistics.groovy core/src/main/resources/hudson/tasks/BuildTrigger/config.jelly core/src/main/resources/hudson/tasks/Fingerprinter/FingerprintAction/index.jelly core/src/main/resources/hudson/tasks/Maven/config.jelly core/src/main/resources/hudson/tasks/junit/CaseResult/index.jelly core/src/main/resources/hudson/tasks/junit/ClassResult/body.jelly core/src/main/resources/hudson/tasks/test/AbstractTestResultAction/summary.jelly core/src/main/resources/hudson/tasks/test/MetaTabulatedResult/body.jelly core/src/main/resources/hudson/tools/JDKInstaller/config.jelly core/src/main/resources/hudson/tools/label.jelly core/src/main/resources/hudson/views/JobColumn/column.jelly core/src/main/resources/jenkins/diagnostics/SecurityIsOffMonitor/message_fr.properties core/src/main/resources/jenkins/management/Messages_pl.properties core/src/main/resources/jenkins/management/PluginsLink/info_pl.properties core/src/main/resources/jenkins/model/CauseOfInterruption/UserInterruption/summary_pl.properties core/src/main/resources/jenkins/model/Jenkins/legend_fr.properties core/src/main/resources/jenkins/model/Jenkins/manage.jelly core/src/main/resources/jenkins/mvn/GlobalMavenConfig/config.groovy core/src/main/resources/jenkins/widgets/BuildQueueWidget/index.groovy core/src/main/resources/jenkins/widgets/ExecutorsWidget/index.groovy core/src/main/resources/lib/form/apply_fr.properties core/src/main/resources/lib/form/booleanRadio_fr.properties core/src/main/resources/lib/form/breadcrumb-config-outline_fr.properties core/src/main/resources/lib/form/dropdownDescriptorSelector.jelly core/src/main/resources/lib/form/hetero-list_fr.properties core/src/main/resources/lib/form/number.jelly core/src/main/resources/lib/form/textarea.jelly core/src/main/resources/lib/hudson/buildListTable.jelly core/src/main/resources/lib/hudson/executors.jelly core/src/main/resources/lib/hudson/jobLink.jelly core/src/main/resources/lib/hudson/queue.jelly core/src/main/resources/lib/layout/breadcrumbBar_fr.properties core/src/main/resources/lib/layout/breadcrumbs.js core/src/main/resources/lib/layout/breakable.jelly core/src/main/resources/lib/layout/copyButton/copyButton.js core/src/main/resources/lib/layout/layout.jelly core/src/main/resources/lib/layout/layout_fr.properties core/src/main/resources/lib/layout/progressiveRendering.jelly core/src/main/resources/lib/layout/progressiveRendering/progressiveRendering.js core/src/main/resources/lib/layout/progressiveRendering_fr.properties core/src/main/resources/lib/layout/task_fr.properties core/src/main/resources/windows-service/jenkins-slave.xml core/src/main/resources/windows-service/jenkins.xml core/src/test/java/hudson/FilePathTest.java core/src/test/java/hudson/FunctionsTest.java core/src/test/java/hudson/logging/LogRecorderTest.java core/src/test/java/hudson/markup/MyspacePolicyTest.java core/src/test/java/hudson/matrix/CombinationFilterUsingBuildParamsTest.java core/src/test/java/hudson/model/FingerprintTest.java core/src/test/java/hudson/model/ListViewTest.java core/src/test/java/hudson/model/StubJob.java core/src/test/java/hudson/tasks/junit/SuiteResultTest.java core/src/test/java/hudson/util/CopyOnWriteMapTest.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java core/src/test/java/jenkins/model/lazy/AbstractLazyLoadRunMapTest.java core/src/test/resources/hudson/model/fingerprint.xml debian/debian/changelog maven-plugin/pom.xml maven-plugin/src/main/java/hudson/maven/AbstractMavenProcessFactory.java maven-plugin/src/main/java/hudson/maven/AggregatingClassLoader.java maven-plugin/src/main/java/hudson/maven/Maven3Builder.java maven-plugin/src/main/java/hudson/maven/Maven3ProcessFactory.java maven-plugin/src/main/java/hudson/maven/MavenBuildInformation.java maven-plugin/src/main/java/hudson/maven/MavenEmbedderRequest.java maven-plugin/src/main/java/hudson/maven/MavenModuleSet.java maven-plugin/src/main/java/hudson/maven/MavenModuleSetBuild.java maven-plugin/src/main/java/hudson/maven/MavenProbeAction.java maven-plugin/src/main/java/hudson/maven/MavenProcessFactory.java maven-plugin/src/main/java/hudson/maven/MavenUtil.java maven-plugin/src/main/java/hudson/maven/PlexusModuleContributor.java maven-plugin/src/main/java/hudson/maven/reporters/MavenAbstractArtifactRecord.java maven-plugin/src/main/java/hudson/maven/reporters/MavenFingerprinter.java maven-plugin/src/main/java/hudson/maven/reporters/TestMojo.java maven-plugin/src/main/resources/hudson/maven/MavenBuild/executedMojos_fr.properties maven-plugin/src/main/resources/hudson/maven/MavenModuleSetBuild/main_pl.properties maven-plugin/src/main/resources/hudson/maven/Messages.properties maven-plugin/src/main/resources/hudson/maven/Messages_es.properties maven-plugin/src/main/resources/hudson/maven/Messages_zh_TW.properties maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_fr.properties maven-plugin/src/test/java/hudson/maven/reporters/SurefireArchiverUnitTest.java maven-plugin/src/test/java/hudson/maven/reporters/TestMojoTest.java plugins/pom.xml pom.xml test/pom.xml test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java test/src/main/java/org/jvnet/hudson/test/MemoryAssert.java test/src/main/java/org/jvnet/hudson/test/MockFolder.java test/src/main/java/org/jvnet/hudson/test/RunLoadCounter.java test/src/main/resources/org/jvnet/hudson/test/ComputerConnectorTester/configure_fr.properties test/src/test/groovy/hudson/cli/BuildCommandTest.groovy test/src/test/groovy/hudson/cli/SetBuildParameterCommandTest.groovy test/src/test/java/hudson/PluginManagerTest.java test/src/test/java/hudson/cli/CopyJobCommandTest.java test/src/test/java/hudson/init/impl/GroovyInitScriptTest.java test/src/test/java/hudson/logging/LogRecorderManagerTest.java test/src/test/java/hudson/matrix/MatrixTest.java test/src/test/java/hudson/maven/MavenModuleSetTest.java test/src/test/java/hudson/maven/MavenMultiModuleLogRotatorCleanArtifactsTest.java test/src/test/java/hudson/maven/MavenMultiModuleTest.java test/src/test/java/hudson/maven/MavenProjectTest.java test/src/test/java/hudson/maven/PlexusModuleContributorTest.java test/src/test/java/hudson/model/AbstractProjectTest.java test/src/test/java/hudson/model/DisplayNameTest.java test/src/test/java/hudson/model/JobTest.java test/src/test/java/hudson/model/ListViewTest.java test/src/test/java/hudson/model/ParametersTest.java test/src/test/java/hudson/model/RunParameterDefinitionTest.java test/src/test/java/hudson/model/RunTest.java test/src/test/java/hudson/tasks/FingerprinterTest.java test/src/test/java/hudson/tasks/MavenTest.java test/src/test/java/hudson/tasks/junit/JUnitResultArchiverTest.java test/src/test/java/hudson/tasks/junit/SuiteResultTest.java test/src/test/java/hudson/tasks/junit/TestNameTransformerTest.java test/src/test/java/hudson/tasks/junit/TestResultPublishingTest.java test/src/test/java/jenkins/model/JenkinsTest.java test/src/test/java/jenkins/security/RekeySecretAdminMonitorTest.java test/src/test/java/org/jvnet/hudson/main/UseRecipesWithJenkinsRuleTest.java test/src/test/resources/hudson/init/impl/GroovyInitScriptTest/errorsHandled.zip test/src/test/resources/hudson/maven/custom-plexus-component.pom test/src/test/resources/hudson/model/JobTest/configDotXmlPermission.zip test/src/test/resources/hudson/model/JobTest/getArtifactsUpTo.zip test/src/test/resources/hudson/model/JobTest/readPermission.zip test/src/test/resources/hudson/model/JobTest/testConfigDotXmlPermission.zip test/src/test/resources/hudson/model/JobTest/testGetArtifactsUpTo.zip test/src/test/resources/hudson/model/JobTest/testReadPermission.zip test/src/test/resources/hudson/model/ListViewTest/nullJobNames/config.xml test/src/test/resources/hudson/tasks/FingerprinterTest/actionSerialization.zip ui-samples-plugin/pom.xml ui-samples-plugin/src/main/resources/index.jelly war/pom.xml war/src/main/webapp/css/style.css war/src/main/webapp/help/parameter/file.html war/src/main/webapp/help/parameter/run-filter.html war/src/main/webapp/help/parameter/run-project.html war/src/main/webapp/help/project-config/description.html war/src/main/webapp/help/run-config/description.html war/src/main/webapp/help/system-config/systemMessage.html war/src/main/webapp/help/user/description.html war/src/main/webapp/help/view-config/description.html war/src/main/webapp/scripts/hudson-behavior.js http://jenkins-ci.org/commit/jenkins/dedb24ef034a04fea288eaa50089e8e774714e99 Log: Merge branch 'master' into JENKINS-14362-jzlib

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Jesse Glick Path: changelog.html

core/pom.xml core/src/main/java/hudson/ExtensionListView.java core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/Functions.java core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/TcpSlaveAgentListener.java core/src/main/java/hudson/Util.java core/src/main/java/hudson/cli/CliProtocol.java core/src/main/java/hudson/cli/ListJobsCommand.java core/src/main/java/hudson/model/AbstractBuild.java core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/model/ComputerSet.java core/src/main/java/hudson/model/Executor.java core/src/main/java/hudson/model/Fingerprint.java core/src/main/java/hudson/model/Label.java core/src/main/java/hudson/model/Node.java core/src/main/java/hudson/model/ParametersDefinitionProperty.java core/src/main/java/hudson/model/Queue.java core/src/main/java/hudson/model/RestartListener.java core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/model/Slave.java core/src/main/java/hudson/model/View.java core/src/main/java/hudson/model/listeners/RunListener.java core/src/main/java/hudson/model/queue/AbstractQueueTask.java core/src/main/java/hudson/model/queue/CauseOfBlockage.java core/src/main/java/hudson/model/queue/MappingWorksheet.java core/src/main/java/hudson/model/queue/QueueListener.java core/src/main/java/hudson/model/queue/SubTask.java core/src/main/java/hudson/model/queue/SubTaskContributor.java core/src/main/java/hudson/model/queue/Tasks.java core/src/main/java/hudson/node_monitors/AbstractDiskSpaceMonitor.java core/src/main/java/hudson/node_monitors/AbstractNodeMonitorDescriptor.java core/src/main/java/hudson/node_monitors/DiskSpaceMonitor.java core/src/main/java/hudson/node_monitors/DiskSpaceMonitorDescriptor.java core/src/main/java/hudson/node_monitors/NodeMonitor.java core/src/main/java/hudson/node_monitors/TemporarySpaceMonitor.java core/src/main/java/hudson/security/GlobalSecurityConfiguration.java core/src/main/java/hudson/slaves/CloudProvisioningListener.java core/src/main/java/hudson/slaves/NodePropertyDescriptor.java core/src/main/java/hudson/slaves/NodeProvisioner.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/hudson/tasks/junit/CaseResult.java core/src/main/java/hudson/tasks/junit/ClassResult.java core/src/main/java/hudson/tasks/junit/PackageResult.java core/src/main/java/hudson/util/CopyOnWriteList.java core/src/main/java/hudson/util/PersistedList.java core/src/main/java/jenkins/model/Jenkins.java core/src/main/java/jenkins/security/QueueItemAuthenticator.java core/src/main/java/jenkins/security/QueueItemAuthenticatorConfiguration.java core/src/main/java/jenkins/security/QueueItemAuthenticatorDescriptor.java core/src/main/java/jenkins/slaves/JnlpSlaveAgentProtocol.java core/src/main/java/jenkins/util/TimeDuration.java core/src/main/resources/hudson/matrix/Messages_de.properties core/src/main/resources/hudson/model/AbstractProject/sidepanel.jelly core/src/main/resources/hudson/model/Messages.properties core/src/main/resources/hudson/model/Messages_de.properties core/src/main/resources/hudson/security/Messages_de.properties core/src/main/resources/hudson/views/BuildButtonColumn/column.jelly core/src/main/resources/jenkins/model/GlobalNodePropertiesConfiguration/config.groovy core/src/main/resources/jenkins/security/QueueItemAuthenticator/config.groovy core/src/main/resources/jenkins/security/QueueItemAuthenticatorConfiguration/config.groovy core/src/main/resources/lib/hudson/projectView.jelly core/src/main/resources/lib/layout/confirmationLink.jelly core/src/main/resources/lib/layout/task.jelly core/src/test/java/hudson/LauncherTest.java core/src/test/java/hudson/cli/ListJobsCommandTest.java core/src/test/java/hudson/model/ViewTest.java core/src/test/java/hudson/tasks/junit/ClassResultTest.java debian/debian/changelog maven-plugin/src/main/resources/hudson/maven/Messages.properties maven-plugin/src/main/resources/hudson/maven/Messages_de.properties maven-plugin/src/main/resources/hudson/maven/RedeployPublisher/config_de.properties maven-plugin/src/main/resources/hudson/maven/reporters/MavenAbstractArtifactRecord/index_de.properties pom.xml test/src/main/java/org/jvnet/hudson/test/HudsonTestCase.java test/src/main/java/org/jvnet/hudson/test/JenkinsRule.java test/src/main/java/org/jvnet/hudson/test/TestPluginManager.java test/src/test/java/hudson/maven/MavenMultiModuleTest.java test/src/test/java/hudson/model/QueueTest.java test/src/test/java/hudson/model/ViewTest.java http://jenkins-ci.org/commit/jenkins/b7e2d6127f77711681113a020681508d57622ea5

Log: Merge branch 'master' into JENKINS-14362-jzlib

Conflicts: core/pom.xml

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Jesse Glick Path:

core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/console/ConsoleNote.java core/src/main/java/hudson/model/Run.java core/src/main/java/hudson/model/UsageStatistics.java core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java core/src/main/java/hudson/util/CompressedFile.java

http://jenkins-ci.org/commit/jenkins/59ff3514532f9c7faef2bc6d841aba53a8107f0f Log: Merge pull request #804 from jglick/JENKINS-14362-jzlib Use jzlib Compare: https://github.com/jenkinsci/jenkins/compare/c1c6475bbb1a...59ff3514532f

[scm_issue_link@java.net (JIRA)]

SCM/JIRA link daemon commented on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Code changed in jenkins User: Jesse Glick Path:

changelog.html http://jenkins-ci.org/commit/jenkins/fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332 Log: JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk.

[jglick@cloudbees.com (JIRA)]

Jesse Glick resolved JENKINS-14362 as Fixed

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Fix expected in 1.520. Change By: Jesse Glick (19/Jun/13 12:41 PM) Status: In Progress Resolved Resolution: Fixed

[dogfood@java.net (JIRA)]

dogfood commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Integrated in jenkins_main_trunk #2633 JENKINS-14362 Switch to jzlib for GZIP streaming. (Revision 3701a327a75d35e5d0e5795341cc55bb123a2007) JENKINS-17713 [FIXED JENKINS-14362] These fixes were (mistakenly?) recorded for 1.518 rather than for trunk. (Revision fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332) Result = SUCCESS Jesse Glick : 3701a327a75d35e5d0e5795341cc55bb123a2007 Files : core/src/main/java/hudson/security/HudsonAuthenticationEntryPoint.java core/src/main/java/hudson/model/Run.java war/pom.xml core/src/main/java/hudson/util/CompressedFile.java ui-samples-plugin/pom.xml pom.xml plugins/pom.xml core/pom.xml cli/pom.xml core/src/main/java/hudson/FilePath.java core/src/main/java/hudson/console/AnnotatedLargeText.java core/src/main/java/hudson/model/UsageStatistics.java core/src/main/java/hudson/console/ConsoleNote.java maven-plugin/pom.xml test/pom.xml Jesse Glick : fbba0cc01eeb1c9d7c412a6e7b76c588cd36c332 Files : changelog.html

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.509.2.JENKINS-14362-jzlib/jenkins-war-1.509.2.JENKINS-14362-jzlib.war now also available if anyone using LTS thinks they are running into this.

[justin@harringa.com (JIRA)]

Justin Harringa commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Is there any chance that this might get back-ported besides just providing the WAR? I noticed that you already marked it as an lts-candidate but it didn't make it into 1.509.2 and wasn't rejected either.

[justin@harringa.com (JIRA)]

Justin Harringa edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Is there any chance that this might get backported besides just providing the WAR? I noticed that you already marked it as an lts-candidate but it didn't make it into 1.509.2 and wasn't rejected either.

[justin@harringa.com (JIRA)]

Justin Harringa edited a comment on JENKINS-14362 100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Is there any chance that this might get backported besides just providing the WAR? I noticed that you already marked it as an lts-candidate but it didn't make it into 1.509.2 and wasn't rejected either. Our infra guys prefer installing via RPM.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@justinharringa: not much made it into 1.509.2, but yes this is a candidate for 1.509.3. The most helpful thing you can do to push it forward is to verify that the fix actually works. Since I have never personally observed the original bug I cannot do that, and we are loath to backport unverified fixes, though I have gotten anecdotal reports that it works. Verifying the fix would mean confirming in a comment in this issue that (a) you consistently observed this problem (meaning heavy CPU load with thread dumps showing one or more threads stuck in Deflater.deflateBytes) in an earlier build, such as 1.519 or 1.509.2, but (b) you have not seen it recur since updating to a build with the purported fix (1.520 or 1.509.2.JENKINS-14362-jzlib).

[justin@harringa.com (JIRA)]

Justin Harringa commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

That sounds good. I'll see what I can do. I just reproduced the issue by opening the Wall Display plugin report in a few tabs. After that, I hit refresh rapidly on one of the tabs. Now the CPU is pegged. I'm currently on 1.480.3. I'll try to reproduce on 1.509.2 as well and then move to your patched WAR in this sandbox environment.

[justin@harringa.com (JIRA)]

Justin Harringa edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

That sounds good. I'll see what I can do. I thought I had reproduced the issue by opening the Wall Display plugin report in a few tabs. After that, I hit refresh rapidly on one of the tabs. Then, the CPU was pegged. Unfortunately, the CPU usage calmed down after a while. This seemed to cause issues in one of our more heavily used instances but it appears it corrects itself in the sandbox environment so it could be a red herring.

I'm currently on 1.480.3. I'll try to reproduce on 1.509.2 as well and then move to your patched WAR in this sandbox environment.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

the CPU was pegged Remember that this by itself does not necessarily, or even probably, indicate that you are seeing JENKINS-14362; there are numerous other bugs that could cause inappropriate heavy CPU usage. You have to check the master thread dump for threads using Deflater.deflateBytes. In reports I have heard, the typical symptom is that Jenkins starts off running fine, but over the course of a week or so more and more threads get “stuck” in this method (seemingly at random), each consuming 100% of one CPU, so that eventually there are no free processors left to do anything.

[justin@harringa.com (JIRA)]

Justin Harringa updated JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Sorry, I should have clarified that I didn't see any threads using Deflater.deflateBytes in the my attempt to reproduce using the Wall Display plugin. I have seen that, in conjunction with consistently pegged CPUs, in my production instance on 1.480.3 (see thread-dump-prod.txt). I was hoping to properly indicate that the Wall Display method I was using wouldn't adequately reproduce the situation since the CPU usage fell. Just to make sure I've got it clear, my impression is that the combination of pegged CPUs, which stay pegged, as well as a threadDump with threads using Deflater.deflateBytes is what we're looking for in order to call it a valid reproduction of the problem. Is that correct? It seems like the closest I've gotten to reproducing this is by using dave catalan's suggested steps. Unfortunately, I was only able to get the thread dump with threads using Deflater.deflateBytes but the CPU usage only spiked to 100% for a few minutes. It seems that this would indicate that I haven't reproduced the issue that could be fixed. I attached 3 separate dumps for the reproduced items (thread-dump-reproduce-1.txt appears to be before or after the deflateBytes call). It would seem that we don't have a deterministic way to reproduce this yet. Change By: Justin Harringa (23/Jul/13 10:45 PM) Attachment: thread-dump-reproduce-2.txt Attachment: thread-dump-reproduce-3.txt Attachment: thread-dump-prod.txt Attachment: thread-dump-reproduce-1.txt

[justin@harringa.com (JIRA)]

Justin Harringa edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

That sounds good. I'll see what I can do. I thought I had reproduced the issue by opening the Wall Display plugin report in a few tabs. After that, I hit refresh rapidly on one of the tabs. Then, the CPU was pegged. Unfortunately, the CPU usage calmed down after a while and I didn't see a thread dump indicating the usage of Deflater.deflateBytes. This seemed to cause issues in one of our more heavily used instances but it appears it corrects itself in the sandbox environment so it could be a red herring. This doesn't appear to be an effective way to reproduce the issue. I'm currently on 1.480.3. Once I find a way to reproduce, I'll try to reproduce on 1.509.2 as well and then move to your patched WAR in this sandbox environment.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

the combination of pegged CPUs, which stay pegged, as well as a threadDump with threads using Deflater.deflateBytes is what we're looking for in order to call it a valid reproduction of the problem. Is that correct? Correct. Or at least the other reports I have heard indicate that the threads suffering the problem do not fix themselves spontaneously; but perhaps sometimes they do. Merely having some threads running deflateBytes does not definitely indicate this bug, because you might genuinely be trying to compress a lot of data at that moment.

[dennys.hsieh@gmail.com (JIRA)]

Dennys Hsieh commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Does this version is recommand for production use? http://repo.jenkins-ci.org/releases/org/jenkins-ci/main/jenkins-war/1.518.JENKINS-14362-jzlib/jenkins-war-1.518.JENKINS-14362-jzlib.war We are still use 1.512 now, and we want to upgrade, but I check the community rating, since 1.519, lots of people rollback to previous version. If the patch for 1.518 is ready for production use? We want to use it first to fix the 100% CPU problem for now and wait a stable version in the future.

[jglick@cloudbees.com (JIRA)]

Jesse Glick commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

You are recommended to just use 1.520 or later.

[jglick@cloudbees.com (JIRA)]

Jesse Glick edited a comment on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

You are recommended to just use 1.520 or later. 1.518.JENKINS-14362-jzlib was offered for experimental purposes only and should no longer be used.

[justin@harringa.com (JIRA)]

Justin Harringa commented on JENKINS-14362

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

@Jesse: On Monday, we're going to upgrade from 1.480.3 to the 1.509.2.JENKINS-14362-jzlib.war on an instance that seems to hit this every few days or so. I'll be sure to report by the end of next week if we see this issue crop up again. I'm hopeful that this will help and I suspect reporting the result will be helpful to you.

[audrey.azra@ericsson.com (JIRA)]

Audrey Azra assigned JENKINS-14362 to Audrey Azra

100% CPU load during org.kohsuke.stapler.compression.CompressionFilter.reportException

Change By: Audrey Azra (03/Feb/14 3:37 PM) Assignee: Jesse Glick Audrey Azra