Very slow tomcat/jenkins restart/startup times

[Maven User]

Hi all -

I'm investigating a very slow tomcat/jenkins startup and was wondering
if anyone had any suggestions.

It's a windows 2003 (R2) x64 server (amd64/64 (4 cores)) with 16 gb of
memory with tomcat (6.0.29) and jenkins (1.414) running with jdk 1.6
(1.6.0_17-b04) with the server JVM as reported by the monitoring
plugin:

JVM: Java HotSpot(TM) 64-Bit Server VM, 14.3-b01, mixed mode

I can see that all four cores are at anywhere between 8 - 10% utilized
during startup.

Within the java configuration tab of the Tomcat Service configuration
panel, we've allocated 4+gb to Tomcat as well as have set PermSize
(256m) and MaxPermSize (512m).

Our main (master) node is configured to have zero executors, so it's
essentially just running Tomcat/Jenkins (we also have nexus running on
the same machine).

We have maybe 100 - 150 jobs, most don't poll svn but are down stream
builds but have their own logs, history, etc.

It takes nearly 20 - 30 min to restart and every operation is
painfully slow.

Are there any additional common tomcat/jenkins tuning options that are
used? I've searched around and tweaked things only slightly, but
wouldn't mind some community feedback.

I'm also seeing this massive stacktrace - not sure where this is
coming from just yet:

Jun 27, 2011 12:09:10 PM org.apache.catalina.core.StandardWrapperValve
invoke
SEVERE: Servlet.service() for servlet Stapler threw exception
java.io.IOException
at
org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer(InternalAprOutputBuffer.java:
696)
at org.apache.coyote.http11.InternalAprOutputBuffer
$SocketOutputBuffer.doWrite(InternalAprOutputBuffer.java:726)
at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:
126)
at
org.apache.coyote.http11.InternalAprOutputBuffer.doWrite(InternalAprOutputBuffer.java:
532)
at org.apache.coyote.Response.doWrite(Response.java:560)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:
353)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:
432)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:347)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:
381)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:
370)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:
89)
at
net.bull.javamelody.FilterServletOutputStream.write(FilterServletOutputStream.java:
70)
at
net.bull.javamelody.CounterResponseStream.write(CounterResponseStream.java:
83)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.write(Unknown Source)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker
$1.write(DefaultScriptInvoker.java:93)
at sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
at sun.nio.cs.StreamEncoder.implFlush(Unknown Source)
at sun.nio.cs.StreamEncoder.flush(Unknown Source)
at java.io.OutputStreamWriter.flush(Unknown Source)
at java.io.BufferedWriter.flush(Unknown Source)
at org.dom4j.io.XMLWriter.flush(XMLWriter.java:276)
at org.apache.commons.jelly.XMLOutput.flush(XMLOutput.java:127)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:272)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:
63)
at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:
161)
at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:46)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:
161)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:
38)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:
161)
at
org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:
150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:
161)
at org.apache.commons.jelly.tags.core.WhenTag.doTag(WhenTag.java:46)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:
161)
at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:
38)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:
63)
at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.CallTagLibScript
$1.run(CallTagLibScript.java:98)
at
org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:
91)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:270)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary
$1.run(ReallyStaticTagLibrary.java:99)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at
org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:
119)
at org.apache.commons.jelly.tags.core.CoreTagLibrary
$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:
63)
at
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:
63)
at
org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:
53)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:
90)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:233)
at
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:
53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:233)
at
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:
53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:233)
at
org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:
53)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:561)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:646)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:477)
at org.kohsuke.stapler.Stapler.service(Stapler.java:159)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at hudson.util.PluginServletFilter
$1.doFilter(PluginServletFilter.java:94)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:
157)
at
net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:
131)
at
net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:
82)
at
org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:
84)
at hudson.util.PluginServletFilter
$1.doFilter(PluginServletFilter.java:97)
at
hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:
52)
at hudson.util.PluginServletFilter
$1.doFilter(PluginServletFilter.java:97)
at
hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:
65)
at hudson.util.PluginServletFilter
$1.doFilter(PluginServletFilter.java:97)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:
86)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:84)
at
hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:
51)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:
166)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:
125)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:
142)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:
271)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:
173)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:
249)
at
hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:
66)
at hudson.security.ChainedServletFilter
$1.doFilter(ChainedServletFilter.java:87)
at
hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:
76)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at
hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:
81)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:
235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:
206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:
233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:
191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:
470)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:
127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:
102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:
109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:
298)
at
org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:
861)
at org.apache.coyote.http11.Http11AprProtocol
$Http11ConnectionHandler.process(Http11AprProtocol.java:579)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:
1584)
at java.lang.Thread.run(Unknown Source)

[Nord, James]

There is no magic bullet but I would suggest you look at disk i/o - basically use the fastest possible storage you can (can you pop in an SSD - you can always add a Hudson job to backup the config data to more reliable storage every night)

Other than that - reduce the number of old builds you keep around for each job.

/James

Hi all -

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************