CPU pegged in stapler - zip depflater

[swerner]

Hi,

My jenkins instance was a little slower to respond this morning, I thought maybe a bunch of jobs were running at the time, but the run queue was empty. top showed the process running at 700%, basically 7 threads pegging a core. Each was hung in the following code:

"RequestHandlerThread[#112]" daemon prio=10 tid=0x00002aaab80c7000 nid=0x721e runnable [0x00000000413c8000]
   java.lang.Thread.State: RUNNABLE
        at java.util.zip.Deflater.deflateBytes(Native Method)
        at java.util.zip.Deflater.deflate(Deflater.java:306)
        - locked <0x00000000e194f3c0> (a java.util.zip.ZStreamRef)
        at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:159)
        at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
        at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
        - locked <0x00000000e194f5e8> (a java.util.zip.GZIPOutputStream)
        at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
        at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
        at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
        - locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
        at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
        at java.io.PrintWriter.close(PrintWriter.java:295)
        - locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
        at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
        at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
        at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

I thought maybe I had 7 users streaming job output, but netstat showed no open connections.

The following error was in the log 7 times:
WARNING: Untrapped servlet exception
winstone.ClientSocketException: Failed to write to client
        at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
        at winstone.WinstoneOutputStream.flush(WinstoneOutputStream.java:219)
        at winstone.WinstoneOutputStream.close(WinstoneOutputStream.java:229)
        at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:149)
        at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
        at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:422)
        at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
        at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
        at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
        at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
        at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
        at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
        at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
        at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
        ... 42 more

I can try upgrading to the latest, I'm currently running 1.477, not that far behind, and didn't think stapler changed that much