DSpace 5.9 not coming back up after installing (and then rolling back) Yum updates on RedHat Linux server
Bouchard, Kerry
Yesterday, we tried installing about 200 current yum updates on the RedHat Linux server where we’re running DSpace 5.9, and we’ve not been able to get DSpace to come back up since then. The problem persisted after we rolled back the yum updates and rebooted the server again. The messages in the DSpace log after restarting the dspace service and then the tomcat service seem to point to a problem initiating the Oracle connection, but our Oracle admin has not been able to locate the problem. I am able to make ODBC connections to the DSpace oracle database from another computer, so we know it’s running.
Below is what I see in the dspace log; putting the log in DEBUG mode does not produce any more information. Does anyone know what we should be looking for to pinpoint the source of the problem?
Thanks, Kerry
2021-03-06 08:39:24,652 ERROR org.dspace.storage.rdbms.DatabaseManager @ SQL getDataSource Error -
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:743)
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:666)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:566)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.commons.dbcp.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:75)
at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:974)
at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at org.dspace.storage.rdbms.DatabaseManager.initialize(DatabaseManager.java:1345)
at org.dspace.storage.rdbms.DatabaseManager.getDataSource(DatabaseManager.java:635)
at org.dspace.storage.rdbms.DatabaseManager.getConnection(DatabaseManager.java:614)
at org.dspace.core.Context.init(Context.java:121)
at org.dspace.core.Context.<init>(Context.java:95)
at org.dspace.app.util.AbstractDSpaceWebapp.deregister(AbstractDSpaceWebapp.java:97)
at org.dspace.app.util.DSpaceContextListener.contextDestroyed(DSpaceContextListener.java:146)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:5174)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5838)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:221)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:149)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:899)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:875)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:679)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1966)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:470)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:506)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:595)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:230)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1452)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:496)
... 30 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
... 35 more
DSpace service status immediately after restart:
Redirecting to /bin/systemctl status dspace.service
● dspace.service - Oracle databases service
Loaded: loaded (/usr/lib/systemd/system/dspace.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2021-03-06 09:16:51 CST; 9s ago
Process: 18568 ExecStop=/opt/app/oracle/product/12.2.0/dbhome_1/bin/dbshut /opt/app/oracle/product/12.2.0/dbhome_1 (code=exited, status=0/SUCCESS)
Main PID: 20793 (dbstart)
Tasks: 47
CGroup: /system.slice/dspace.service
├─20793 /bin/sh /opt/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /opt/app/oracle/product/12.2.0/dbhome_1
├─20804 /opt/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr LISTENER -inherit
├─20842 /bin/sh /opt/app/oracle/product/12.2.0/dbhome_1/bin/dbstart /opt/app/oracle/product/12.2.0/dbhome_1
├─20895 sqlplus
├─20908 ora_pmon_dspace
├─20910 ora_clmn_dspace
├─20912 ora_psp0_dspace
├─20915 ora_vktm_dspace
├─20919 ora_gen0_dspace
├─20921 ora_mman_dspace
├─20925 ora_gen1_dspace
├─20929 ora_diag_dspace
├─20931 ora_ofsd_dspace
├─20935 ora_dbrm_dspace
├─20937 ora_vkrm_dspace
├─20939 ora_svcb_dspace
├─20941 ora_pman_dspace
├─20943 ora_dia0_dspace
├─20945 ora_dbw0_dspace
├─20947 ora_dbw1_dspace
├─20949 ora_dbw2_dspace
├─20951 ora_dbw3_dspace
├─20953 ora_lgwr_dspace
├─20955 ora_ckpt_dspace
├─20957 ora_lg00_dspace
├─20959 ora_smon_dspace
├─20961 ora_lg01_dspace
├─20963 ora_smco_dspace
├─20965 ora_reco_dspace
├─20967 ora_w000_dspace
├─20969 ora_lreg_dspace
├─20971 ora_w001_dspace
├─20973 ora_pxmn_dspace
├─20975 ora_dmon_dspace
├─20979 ora_mmon_dspace
├─20981 ora_mmnl_dspace
├─20983 ora_d000_dspace
├─20985 ora_s000_dspace
├─20987 ora_tmon_dspace
├─21009 oracledspace (LOCAL=NO)
├─21138 oracledspace (LOCAL=NO)
└─21142 oracledspace (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
Mar 06 09:16:51 libdspace03.lib.tcu.edu systemd[1]: Started Oracle databases service.
Mar 06 09:16:51 libdspace03.lib.tcu.edu dbstart[20793]: Processing Database instance "dspace": log file /opt/app/oracle/product/12.2.0/dbhome_1/startup.log
DSpace service status a minute or so after restart:
dspace.service - Oracle databases service
Loaded: loaded (/usr/lib/systemd/system/dspace.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Sat 2021-03-06 09:32:29 CST; 1min 59s ago
Main PID: 1864 (code=exited, status=0/SUCCESS)
Mar 06 09:32:09 libdspace03.lib.tcu.edu systemd[1]: Started Oracle databases service.
Mar 06 09:32:15 libdspace03.lib.tcu.edu dbstart[1864]: Processing Database instance "dspace": log file /opt/app/oracle/product/12.2.0/dbhome_1/startup.log
Mar 06 09:32:22 libdspace03.lib.tcu.edu dbshut[9118]: Processing Database instance "dspace": log file /opt/app/oracle/product/12.2.0/dbhome_1/shutdown.log
Mar 06 09:32:47 libdspace03.lib.tcu.edu systemd[1]: [/usr/lib/systemd/system/dspace.service:7] Failed to parse service restart specifier, ignoring: yes
Mar 06 09:34:25 libdspace03.lib.tcu.edu systemd[1]: [/usr/lib/systemd/system/dspace.service:7] Failed to parse service restart specifier, ignoring: yes
|
|
Kerry Bouchard DIRECTOR OF LIBRARY SYSTEMS TCU LIBRARY TCU BOX 298400 FORT WORTH, TX 76129
|
Bouchard, Kerry
After more testing, I see that I can’t connect to the Oracle port on the server even using “nc”
nc localhost 1521
Ncat: Connection refused.
I’m thinking that on a working server, that wouldn’t be happening?
Thanks, Kerry
Bouchard, Kerry
Apparently starting dspace has been crashing the Oracle listener process when the dspace server starts up. This is what our Oracle manager says he’s seeing today:
“The command service dspace start crashes the database if it is already started. If the database is not started and you run the command it will start the database but the database shuts down immediately. When I start the database manually it stays up and running.”
-Kerry
Mark H. Wood
>
> Apparently starting dspace has been crashing the Oracle listener process when the dspace server starts up. This is what our Oracle manager says he’s seeing today:
>
> “The command service dspace start crashes the database if it is already started. If the database is not started and you run the command it will start the database but the database shuts down immediately. When I start the database manually it stays up and running.”
can show why DSpace is doing something specific that causes the crash.
(If it's possible for a client to do *anything* that causes the DBMS
instance to crash, Oracle should fix that.) One supposes that Oracle
is logging some kind of errors as it goes down....
Possibly someone who is familiar with systemd could spot something
that was damaged by the 'yum update'. The two units may be
interfering with each other somehow. Anybody?
> From: Bouchard, Kerry
> Sent: Saturday, March 6, 2021 4:52 PM
>
>
> nc localhost 1521
> Ncat: Connection refused.
>
> I’m thinking that on a working server, that wouldn’t be happening?
> From: Bouchard, Kerry <k.bou...@tcu.edu<mailto:k.bou...@tcu.edu>>
> Sent: Saturday, March 6, 2021 9:36 AM
>
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:607)
> at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:161)
> at oracle.net.nt.ConnOption.connect(ConnOption.java:159)
> at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:428)
> ... 35 more
DBMS is not listening on that port.
--
Mark H. Wood
Lead Technology Analyst
University Library
Indiana University - Purdue University Indianapolis
755 W. Michigan Street
Indianapolis, IN 46202
317-274-0749
www.ulib.iupui.edu