swerner
unread,Oct 25, 2012, 12:12:48 PM10/25/12Sign in to reply to author
Sign in to forward
You do not have permission to delete messages in this group
Sign in to report message
Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message
to jenkins...@googlegroups.com
Hi,
My jenkins instance was a little slower to respond this morning, I thought maybe a bunch of jobs were running at the time, but the run queue was empty. top showed the process running at 700%, basically 7 threads pegging a core. Each was hung in the following code:
"RequestHandlerThread[#112]" daemon prio=10 tid=0x00002aaab80c7000 nid=0x721e runnable [0x00000000413c8000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Deflater.deflateBytes(Native Method)
at java.util.zip.Deflater.deflate(Deflater.java:306)
- locked <0x00000000e194f3c0> (a java.util.zip.ZStreamRef)
at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:159)
at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
- locked <0x00000000e194f5e8> (a java.util.zip.GZIPOutputStream)
at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:297)
at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:130)
- locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.close(OutputStreamWriter.java:216)
at java.io.PrintWriter.close(PrintWriter.java:295)
- locked <0x00000000e194f640> (a java.io.OutputStreamWriter)
at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:77)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
I thought maybe I had 7 users streaming job output, but netstat showed no open connections.
The following error was in the log 7 times:
WARNING: Untrapped servlet exception
winstone.ClientSocketException: Failed to write to client
at winstone.ClientOutputStream.write(ClientOutputStream.java:41)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at winstone.WinstoneOutputStream.commit(WinstoneOutputStream.java:167)
at winstone.WinstoneOutputStream.flush(WinstoneOutputStream.java:219)
at winstone.WinstoneOutputStream.close(WinstoneOutputStream.java:229)
at java.util.zip.DeflaterOutputStream.close(DeflaterOutputStream.java:149)
at org.kohsuke.stapler.compression.FilterServletOutputStream.close(FilterServletOutputStream.java:36)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:422)
at org.kohsuke.stapler.Stapler.serveStaticResource(Stapler.java:249)
at org.kohsuke.stapler.Stapler.service(Stapler.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at winstone.ClientOutputStream.write(ClientOutputStream.java:39)
... 42 more
I can try upgrading to the latest, I'm currently running 1.477, not that far behind, and didn't think stapler changed that much