DSpace 5.4 - tomcat stuck at org.apache.catalina.startup.Catalina start after edit of robots.txt file

[Kerry Bouchard]

Earlier today, I made a backup copy of our /opt/projects/dspace/webapps/xmlui/static/robots.txt file, added a line to stop crawlers from trying to find our /register page, and then stopped and restarted tomcat. When DSPace didn't come back up, I restored the original version of robots.txt, and stopped and restarted tomcat again. It still won't come up.

• Output of the "service tomcat status" shows:
   tomcat.service - Apache Tomcat Web Application Container
     Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/tomcat.service.d
             └─override.conf
     Active: active (running) since Thu 2022-11-03 14:21:53 CDT; 2s ago
   Main PID: 6272 (java)
      Tasks: 75
     CGroup: /system.slice/tomcat.service
             └─6272 /usr/lib/jvm/jre/bin/java -Xmx32g -Xms32g -XX:MaxPermSize=1024m -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/jav...
  
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: at java.lang.Thread.run(Thread.java:750)
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: Nov 03, 2022 2:21:54 PM org.apache.catalina.startup.HostConfig deployDescriptor
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: INFO: Deployment of configuration descriptor /etc/tomcat/Catalina/localhost/xmlui.xml has finished in 97 ms
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: Nov 03, 2022 2:21:54 PM org.apache.coyote.AbstractProtocol start
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: INFO: Starting ProtocolHandler ["http-bio-8080"]
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: Nov 03, 2022 2:21:54 PM org.apache.coyote.AbstractProtocol start
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: INFO: Starting ProtocolHandler ["ajp-bio-8009"]
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: Nov 03, 2022 2:21:54 PM org.apache.catalina.startup.Catalina start
  Nov 03 14:21:54 libdspace03.lib.tcu.edu server[6272]: INFO: Server startup in 609 ms
  
• The dspace.log and cocoon.log files show no activity since since the first stop and restart of tomcat.
• The /var/log/tomcat/catalina.2022-11-03.log file shows this:
  Nov 03, 2022 2:21:54 PM org.apache.catalina.startup.HostConfig deployDescriptor
  SEVERE: Error deploying configuration descriptor /etc/tomcat/Catalina/localhost/xmlui.xml
  java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [/xmlui]
          at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:903)
          at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
          at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
          at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
          at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:750)
  
  Nov 03, 2022 2:21:54 PM org.apache.catalina.startup.HostConfig deployDescriptor
  INFO: Deployment of configuration descriptor /etc/tomcat/Catalina/localhost/xmlui.xml has finished in 97 ms
  Nov 03, 2022 2:21:54 PM org.apache.coyote.AbstractProtocol start
  INFO: Starting ProtocolHandler ["http-bio-8080"]
  Nov 03, 2022 2:21:54 PM org.apache.coyote.AbstractProtocol start
  INFO: Starting ProtocolHandler ["ajp-bio-8009"]
  Nov 03, 2022 2:21:54 PM org.apache.catalina.startup.Catalina start
  INFO: Server startup in 609 ms
  
• The files in /etc/tomcat/Catalina/localhost haven't been modified since 2015.
• The files in  /opt/projects/dspace/webapps/xmlui/static/ have owner and group "tomcat" and the files in  /etc/tomcat/Catalina/localhost have owner and group "dspace". 
• As far as I know, this part of the configuration hasn't changed for years. 
• Oracle and Oracle listener are up and running, but not showing any connections from DSpace.
• I've rebooted the server, but the problem persists.

Our DSpace 5.4 instance is dead in the water, but the only think I changed before stopping and restarting tomcat was the robots.txt file (which is now back to the original).

Any help will be appriciated.

Thanks, Kerry

[Kerry Bouchard]

I edited the tomcat logging.properties file to change the log fevels from INFO to FINE, and now when I restart tomcat, the first severe error in the  catalina.2022-11-03.log file is:

INFO: Deploying configuration descriptor /etc/tomcat/Catalina/localhost/ROOT.xml
Nov 03, 2022 5:31:14 PM org.apache.tomcat.util.digester.Digester fatalError
SEVERE: Parse Fatal Error at line 1 column 2: The markup declarations contained or pointed to by the document type declaration must be well-formed.
org.xml.sax.SAXParseException; systemId: http://www.eclipse.org/jetty/configure.dtd; lineNumber: 1; columnNumber: 2; The markup declarations contained or pointed to by the document type declaration must be well-formed.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
    [followed by long stack trace]

The /etc/tomcat/Catalina/localhost/ROOT.xml file has not been edited since 2015. For what it's worth, the contents are:

<?xml version="1.0" ?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure.dtd">

<!-- ==================================================================
Configure and deploy the test web application in $(jetty.home)/webapps/test

Note. If this file did not exist or used a context path other that /test
then the default configuration of jetty.xml would discover the test
webapplication with a WebAppDeployer.  By specifying a context in this
directory, additional configuration may be specified and hot deployments
detected.
===================================================================== -->

<!--
<Configure class="org.eclipse.jetty.webapp.WebAppContext">
  <Set name="contextPath">/</Set>
  <Set name="war">/opt/projects/dspace/webapps/xmlui</Set>
</Configure>
-->

<Context
    docBase="/opt/projects/dspace/webapps/xmlui"
    debug="0"
    reloadable="true"
    cachingAllowed="false"
    allowLinking="false"/>

[Mark H. Wood]

On Thu, Nov 03, 2022 at 03:47:16PM -0700, Kerry Bouchard wrote:
> I edited the tomcat logging.properties file to change the log fevels from
> INFO to FINE, and now when I restart tomcat, the first severe error in the
> catalina.2022-11-03.log file is:
>
> INFO: Deploying configuration descriptor
> /etc/tomcat/Catalina/localhost/ROOT.xml
> Nov 03, 2022 5:31:14 PM org.apache.tomcat.util.digester.Digester fatalError
> SEVERE: Parse Fatal Error at line 1 column 2: The markup declarations
> contained or pointed to by the document type declaration must be
> well-formed.
> org.xml.sax.SAXParseException; systemId:
> http://www.eclipse.org/jetty/configure.dtd; lineNumber: 1; columnNumber: 2;
> The markup declarations contained or pointed to by the document type
> declaration must be well-formed.
> at
> com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
> [followed by long stack trace]
>
> The /etc/tomcat/Catalina/localhost/ROOT.xml file has not been edited since
> 2015. For what it's worth, the contents are:
>
> <?xml version="1.0" ?>
> <!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN"

This refers to the DTD for configuration of Jetty, not Tomcat. That
may be confusing the parser. I don't know why it should suddenly
become a problem. Perhaps you upgraded Tomcat recently and the new
version is more strict?

I would remove everything starting at the DOCTYPE element down to but
not including the "<Context" line.

> > - Output of the "service tomcat status" shows:

> > - The dspace.log and cocoon.log files show no activity since since the

> > first stop and restart of tomcat.

> > - The /var/log/tomcat/catalina.2022-11-03.log file shows this:

> > - The files in /etc/tomcat/Catalina/localhost haven't been modified
> > since 2015.
> > - The files in /opt/projects/dspace/webapps/xmlui/static/ have owner

> > and group "tomcat" and the files in /etc/tomcat/Catalina/localhost have
> > owner and group "dspace".

> > - As far as I know, this part of the configuration hasn't changed for
> > years.
> > - Oracle and Oracle listener are up and running, but not showing any
> > connections from DSpace.
> > - I've rebooted the server, but the problem persists.

> >
> > Our DSpace 5.4 instance is dead in the water, but the only think I changed
> > before stopping and restarting tomcat was the robots.txt file (which is now
> > back to the original).
> >
> > Any help will be appriciated.
> >
> > Thanks, Kerry
> >
> >
> >
>

> --
> All messages to this mailing list should adhere to the Code of Conduct: https://www.lyrasis.org/about/Pages/Code-of-Conduct.aspx
> ---
> You received this message because you are subscribed to the Google Groups "DSpace Technical Support" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dspace-tech...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dspace-tech/a91aa2fa-2bd2-4f1d-9ddf-f491b99ee07bn%40googlegroups.com.


--
Mark H. Wood
Lead Technology Analyst

University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu

[Bouchard, Kerry]

Mark,
THANK YOU SO MUCH!!! I did that edit for all the .XML files in our /etc/tomcat/Catalina/localhost directory, and that got us up and running again.

Thanks again, Kerry

-----Original Message-----
From: Mark H. Wood,UL 0115A,+1 317 274 0749, <mw...@IUPUI.Edu> On Behalf Of Mark H. Wood
Sent: Friday, November 4, 2022 7:23 AM
To: dspac...@googlegroups.com
Subject: Re: [dspace-tech] Re: DSpace 5.4 - tomcat stuck at org.apache.catalina.startup.Catalina start after edit of robots.txt file

[EXTERNAL EMAIL WARNING] DO NOT CLICK LINKS or open attachments unless you recognize the sender and know the content is safe.