Super slow startup after Tomcat 8 update

[Matthew Jones]

So after updating my local build to tomcat 8, the startups got *really* slow, like 4 times slower. This is on a newer i7 and SSD that was getting ~100000 ms startup times on Tomcat 7.

Previously there was a catalina.property suggested [1] that improved startup times

org.apache.catalina.startup.ContextConfig.jarsToSkip=*.jar

But it seemed like there is some more scanning going on in Tomcat 8. The write up here [1] mentioned another property that gets it almost back to normal.

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*.jar

Looks like it starts up and everything is okay. 

I believe we could configure every tool's web.xml to skip and eliminate having to set this [3] just not something that has been done (SAK-22183) since the workaround is so easy.

[1] https://steveswinsburg.wordpress.com/2012/11/06/reduce-your-sakai-cle-startup-time/

[2] http://tonyjunkes.com/blog/fixing-slow-railo-startup-running-on-tomcat-8

[3] http://wiki.apache.org/tomcat/HowTo/FasterStartUp#Configure_your_web_application

[Matthew Jones]

Okay, this suggestion yesterday isn't viable because it looks like every JSF tool is breaking. It seems like to improve startup times in Tomcat 8 we'd need to.

1) Modify every web.xml (looks like over 200 in Sakai core alone) to tell tomcat to skip scanning that app

Set metadata-complete="true" attribute on the <web-app> element.

Add an empty <absolute-ordering /> element.

Or

2) Use the property to skip all apps and register a listener in these JSF apps. There might be more to do after that.

<listener>
  <listener-class>org.apache.myfaces.webapp.StartupServletContextListener</listener-class>
</listener>

3) Come up with a long property to selectively skip some additional jars. I think turning on debug in the logs would list which jars can be safely skipped.

I kind of like #1 but that seems like the most time consuming. But with a startup time that increases from 1 minute to 5 minutes, it would save time in the long run.

[Matthew Buckett]

I just checked this with my local Tomcat 8 install and I'm getting:

23-Apr-2015 15:58:09.827 INFO [main]
org.apache.catalina.startup.Catalina.start Server startup in 92912 ms

*without* your tomcat.util.scan.StandardJarScanFilter setting but with
the one that ships with Tomcat 8.

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=\
bootstrap.jar,commons-daemon.jar,tomcat-juli.jar,\
annotations-api.jar,el-api.jar,jsp-api.jar,servlet-api.jar,websocket-api.jar,\
catalina.jar,catalina-ant.jar,catalina-ha.jar,catalina-storeconfig.jar,\
catalina-tribes.jar,\
jasper.jar,jasper-el.jar,ecj-*.jar,\
tomcat-api.jar,tomcat-util.jar,tomcat-util-scan.jar,tomcat-coyote.jar,\
tomcat-dbcp.jar,tomcat-jni.jar,tomcat-spdy.jar,tomcat-websocket.jar,\
tomcat-i18n-en.jar,tomcat-i18n-es.jar,tomcat-i18n-fr.jar,tomcat-i18n-ja.jar,\
tomcat-juli-adapters.jar,catalina-jmx-remote.jar,catalina-ws.jar,\
tomcat-jdbc.jar,\
tools.jar,\
commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,\
commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,\
commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,\
commons-math*.jar,commons-pool*.jar,\
jstl.jar,\
geronimo-spec-jaxrpc*.jar,wsdl4j*.jar,\
ant.jar,ant-junit*.jar,aspectj*.jar,jmx.jar,h2*.jar,hibernate*.jar,httpclient*.jar,\
jmx-tools.jar,jta*.jar,log4j*.jar,mail*.jar,slf4j*.jar,\
xercesImpl.jar,xmlParserAPIs.jar,xml-apis.jar,\
junit.jar,junit-*.jar,ant-launcher.jar,\
cobertura-*.jar,asm-*.jar,dom4j-*.jar,icu4j-*.jar,jaxen-*.jar,jdom-*.jar,\
jetty-*.jar,oro-*.jar,servlet-api-*.jar,tagsoup-*.jar,xmlParserAPIs-*.jar,\
xom-*.jar


--
Matthew Buckett, VLE Developer, IT Services, University of Oxford

[Matthew Buckett]

I think this is how I'm getting my copy of Tomcat 8 setup:

https://gist.github.com/buckett/ae65055238d143a9d742

> --
> You received this message because you are subscribed to the Google Groups "Sakai Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
> To post to this group, send email to saka...@apereo.org.
> Visit this group at http://groups.google.com/a/apereo.org/group/sakai-dev/.

[Matthew Jones]

Yeah, that's a really good time then. The nightly servers are getting about the same as my local host with the default property.

INFO: Server startup in 342275 ms

Previously on Tomcat 7 (and with the jarsToSkip=*.jar) it was under 100000 ms.

This message below appears 89 times in the logs. There looks to be around a 3 second pause each time on each tool. Thats ~267 seconds alone.

INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

[Brian Jones]

+1 for #1; script to do the monotonous work of modifying every web.xml?

Brian Jones
Programmer/Analyst
Information Technology Services
Support Services Building, Suite 4300
Western University
(519) 661-2111 x86969
bjon...@uwo.ca

> --
> You received this message because you are subscribed to the Google
> Groups "Sakai Development" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to sakai-dev+...@apereo.org

> <mailto:sakai-dev+...@apereo.org>.

> To post to this group, send email to saka...@apereo.org

> <mailto:saka...@apereo.org>.

[Matthew Buckett]

To get performance better and to have the JSF tools working again we
only need todo scanning for JARs that contain tag library definitions.
So locally you can set:

tomcat.util.scan.StandardJarScanFilter.jarsToSkip=*

tomcat.util.scan.StandardJarScanFilter.jarsToScan=log4j-core*.jar,log4j-taglib*.jar,jsf-impl-*.jar,jsf-widgets-*.jar,struts-*.jar,tomahawk-*.jar,spring-webmvc-*.jar,standard-*.jar,myfaces-impl-*.jar,sakai-sections-app-util-*.jar

And from a quick test all tools run.

A much better long term solution is to have each tool say what needs
scanning (as Matt suggested), but this should get people starting up
quickly in the short term.

[Earle Nietzel]

Just one more note on this!

I was able to get the start times back to what they were setting this
in conf/context.xml:

<Context>
<JarScanner>
<JarScanFilter defaultPluggabilityScan="false" />
</JarScanner>
</Context>

JSF tools are working as well. Didn't need to set any other config in
conf/catalina.properties

> --
> You received this message because you are subscribed to the Google Groups "Sakai Development" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sakai-dev+...@apereo.org.
> To post to this group, send email to saka...@apereo.org.

> Visit this group at http://groups.google.com/a/apereo.org/group/sakai-dev/.

--
earle,
asahi net int.

[Brian Jones]

I'm back down to ~1min start-up times with these latest settings from
Earle and Matt B.; very acceptable :)

Brian Jones
Programmer/Analyst
Information Technology Services
Support Services Building, Suite 4300
Western University
(519) 661-2111 x86969
bjon...@uwo.ca

[Matthew Jones]

Yeah, thanks all, those settings really help! Just the context.xml change alone gets it down to a respectable time again.