Opal not starting - opal-config?
Tom Bishop
Hi Yannick,
I hope all is well with you.
Our production Opal instance (running as a container under Coral) failed and won’t restart. The error message seems to indicate an issue with opal-config, perhaps some kind of corruption after an unclean shutdown.
I’m afraid I’ve not seen this before and searching for the error has not turned up a solution. Do you have any suggestions, please?
Thanks
Tom
2024-09-10 12:29:10,703 [main] INFO org.obiba.opal.server.OpalServer - Starting Opal server!
2024-09-10 12:29:10,706 [main] INFO org.obiba.opal.server.OpalServer - Upgrading Opal...
2024-09-10 12:29:10,774 [main] INFO org.springframework.context.support.ClassPathXmlApplicationContext - Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@a9cd3b1: startup date [Tue Sep 10 12:29:10 UTC 2024]; root of context hierarchy
2024-09-10 12:29:10,818 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/spring/opal-server/upgrade.xml]
2024-09-10 12:29:10,898 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [META-INF/spring/opal-server/config.xml]
2024-09-10 12:29:10,927 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [upgrade-context.xml]
2024-09-10 12:29:11,044 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [spring/opal-core/tx.xml]
2024-09-10 12:29:11,069 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from class path resource [spring/opal-core/version.xml]
2024-09-10 12:29:11,397 [main] INFO org.objectweb.jotm - CAROL initialization
2024-09-10 12:29:11,408 [main] INFO org.objectweb.jotm - JOTM started with a local transaction factory
2024-09-10 12:29:11,477 [main] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA UserTransaction: org.objectweb.jotm.Current@6ce86ce1
2024-09-10 12:29:11,478 [main] INFO org.springframework.transaction.jta.JtaTransactionManager - Using JTA TransactionManager: org.objectweb.jotm.Current@6ce86ce1
2024-09-10 12:29:12,156 [main] INFO org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.0.1.Final
2024-09-10 12:29:12,298 [main] INFO org.obiba.opal.core.service.LocalOrientDbServerFactory - Start OrientDB server (plocal:/srv/data/orientdb/opal-config)
2024-09-10 12:29:12,636 [main] INFO com.orientechnologies.orient.server.config.OServerConfigurationLoaderXml - Loading configuration from input stream
2024-09-10 12:29:15,995 [main] ERROR com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Exception `45900B64` in storage `plocal:/srv/data/orientdb/opal-config`: 2.2.37 (build a7541e7ceeabf592dd9a7b2928b6c023cbc73193, branch 2.2.x)
java.lang.IllegalMonitorStateException: null
at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131)
at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.releaseLock(OOneEntryPerKeyLockManager.java:216)
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:493)
at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:407)
at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:113)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.create(OSBTree.java:156)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.create(OSBTreeIndexEngine.java:82)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addIndexEngine(OAbstractPaginatedStorage.java:2048)
at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:243)
at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:129)
at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:40)
at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:171)
at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:87)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1900)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1870)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1860)
at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1856)
at org.obiba.opal.core.service.OrientDbServiceImpl.createUniqueIndex(OrientDbServiceImpl.java:401)
at org.obiba.opal.core.service.security.SubjectCredentialsServiceImpl.start(SubjectCredentialsServiceImpl.java:85)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:366)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:311)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:134)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:407)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1611)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:481)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:756)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
at org.obiba.opal.server.UpgradeCommand.standardUpgrade(UpgradeCommand.java:61)
at org.obiba.opal.server.UpgradeCommand.execute(UpgradeCommand.java:57)
at org.obiba.opal.server.OpalServer.upgrade(OpalServer.java:102)
at org.obiba.opal.server.OpalServer.<init>(OpalServer.java:37)
at org.obiba.opal.server.OpalServer.main(OpalServer.java:151)
Stuart Wheater
opal-1 | Unrecognized VM option 'MaxPermSize=256M'
opal-1 | Error: Could not create the Java Virtual Machine.
opal-1 | Error: A fatal exception has occurred. Program will exit.
opal-1 | Unrecognized VM option 'MaxPermSize=256M'
opal-1 | Error: Could not create the Java Virtual Machine.
opal-1 | Error: A fatal exception has occurred. Program will exit.
Registration No: 4497081
--
You received this message because you are subscribed to the Google Groups "obiba-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obiba-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB3792526D6CC3A00FE333A344B9612%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Yannick Marcon
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAFgjRZp9tEcLF7f4ko8QyU2TQaO6Bpp6hCQzhcnwchB3BOYKsA%40mail.gmail.com.
Stuart Wheater
Registration No: 4497081
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAGrE6ouWmraNEUhw8Jbq1N9D8DuTqx77wKFFexYAnh2tnJyY%2BQ%40mail.gmail.com.
Yannick Marcon
--
Tom Bishop
Sent: 17 September 2024 7:47 PM
To: obiba...@googlegroups.com <obiba...@googlegroups.com>
Subject: Re: [OBiBa] Opal not starting - opal-config?
Stuart Wheater
Registration No: 4497081
Yannick Marcon
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAFgjRZrj6htyd04aM9c0jmyOi4zQ3Fh3vyvSTzxiuLFykhQyPA%40mail.gmail.com.
Tom Bishop
Just to follow up on this, if I don’t think another Opal process is running, my only option is to repair OrientDB?
What is the role of OrientDB? Does it contain information about the configuration of Opal? Should we back it up in the future?
Thanks
Tom
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB3792FA291D2E6E7580741F9CB9622%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Yannick Marcon
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB3792DD90097EB226E2E629ECB9632%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Tom Bishop
Yannick Marcon
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB379246797554D75C34EEE5F8B96C2%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Tom Bishop
Yannick Marcon
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB379243E6F412FD1ACF7FD504B96C2%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Ramin H.A.
Hi Tom,
Try running the repair database and rebuild index as follows:
$ repair database --fix-graph=true --fix-links=true
$ rebuild index *
FYI: https://orientdb.org/docs/2.2.x/Console-Command-Repair-Database.html
Good luck!
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB379243E6F412FD1ACF7FD504B96C2%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Tom Bishop
Thanks Ramin, I tried this and it made no difference. No errors were reported during the repair and reindex.
When running the repair it looks like there is nothing in the database (i.e. no edges or vertices):
Repair of graph 'plocal:/srv/data/orientdb.test/opal-config' is started ...
Scanning 0 edges (skipEdges=0)...
Scanning edges completed
Scanning 0 vertices...
Scanning vertices completed
Repair of graph 'plocal:/srv/data/orientdb.test/opal-config' completed in 0 secs
scannedEdges.....: 0
removedEdges.....: 0
scannedVertices..: 0
scannedLinks.....: 0
removedLinks.....: 0
repairedVertices.: 0
Do you have any other ideas about what might be going on?
Thanks
Tom
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAMpqDgd49M8QL9rd8skYFThQZz%2BT9NrJjcCRZgkXr88cv-UjbQ%40mail.gmail.com.
Ramin H.A.
Hi Tom,
When you run the command ls -al inside the opal-config are the following files all empty? *.cpm, *.cpl. Also, just in case, make sure the ownership is correct. For example, my Opal docker container volumes are owned by systemd-resolve.
Can you do any queries inside the OrientDB such as:
$ classes
--or--
$ select * from Database
$ select * from SubjectAcl
I am afraid the only viable solution looks more like reverting to a functional backup and hopefully the backups are frequent enough that you can revert to a version that is functional as well as recent.
I will see if I can find anything useful.
Best,
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB3792680B569C6EA8C5556079B96F2%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Tom Bishop
Thanks for your help Ramin.
The .cpm and .pcl are not all empty, some have size 66560, 132096 etc.
When I do select commands I can return entries that correspond to projects in our mysql database, for example, so there is definitely “stuff” still there. The ownership in our case is opal and nogroup, which seems consistent with other files in the container.
Unfortunately all the backups are currently locked away (possibly for months due to a cyber attack). I really hope they restore a working version otherwise we have lost a huge amount of work.
Best wishes
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAMpqDgeumnrB%3DUqCQkgXAX3TnaZRaxP2s%2B3eoUzB%3DFdjvO76Zw%40mail.gmail.com.
Ramin H.A.
The good news is that the actual data (datasources, view and identifiers) is not in the OrientDB as this database is primarily for configurations.
If you have time, you could:
- Backup the current volumes to another folder where you’d start a new Opal instance (use
rsync -avPto preserve ownership) - Start a fresh new Opal container with the same version as the previous one
- Configure your new Opal database such that it points to the correct Opal Data and Identifiers DBs (doc), e.g.:
# Old OrientDB
select * from Database
# Output
+----+-----+--------+------------+-------+--------------+------------------+---------------------------------------------------------------------------------------+-------------------+-------------------+
|# |@RID |@CLASS |name |usage |defaultStorage|usedForIdentifiers|mongoDbSettings |created |updated |
+----+-----+--------+------------+-------+--------------+------------------+---------------------------------------------------------------------------------------+-------------------+-------------------+
|0 |#33:0|Database|_identifiers|STORAGE|false |true |{url=mongodb://mongo:27017/opal_ids, username=, password=, properties=, batchSize=100} |2024-08-06 12:02:35|2024-08-06 12:02:35|
|1 |#34:0|Database|mongodb |STORAGE|true |false |{url=mongodb://mongo:27017/opal_data, username=, password=, properties=, batchSize=100}|2024-08-06 12:02:36|2024-08-06 12:02:36|
# New OrientDB
connect plocal:/data/new-volumes/opal/data/orientdb/opal-config admin admin
update Database merge {mongoDbSettings: {url:'mongodb://mongo:27017/opal_data?authSource=admin', username:'toto', password:'ABCDFED', properties:'', batchSize:100}} where name = 'opal_data'
update Database merge {mongoDbSettings: {url:'mongodb://mongo:27017/opal_ids?authSource=admin', username:'toto', password:'JDHEGEE', properties:'', batchSize:100}} where name <> 'opal_data'
Sometimes and not often I use this command to point the Database classes to another IP.
- Stop the new container and copy all relevant volumes from your backup above (work, data, fs, etc) except:
- data/orientdb
- data/keystore
- Make sure all the folders in the volume folder have proper permissions
- Cross fingers
- Start the instance
If Opal starts without any issues, you can try to create a Project you know (In the old OrientDB select * from Project limit 1000). Since the actual data DB still has your datasources, it should be able to recuperate all tables and variables.
What is lost? ACLs, users, groups, alas all configurations but you may able to, painfully, recreating them by finding the required info via the old database, say, to find all old users :
select * from SubjectAcl limit 1000 # use temp password and let the users change them
One last thing, if you have Opal, Mongo/MySQL, Rock all as containers using a docker-compose.yml, recreate them all with the old volumes (make sure they have the same versions) and follow the above suggestions for Opal only.
Best of lucks,
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/LO4P265MB37927EB13E22F88AA361768CB96F2%40LO4P265MB3792.GBRP265.PROD.OUTLOOK.COM.
Tom Bishop
Hi Ramin,
Thanks again for your guidance. I see that there is an option to essentially manually reconstruct the information held in orientdb via the console. I think I will have to wait for the backups first and see if that works.
Best wishes
Tom
From: obiba...@googlegroups.com <obiba...@googlegroups.com>
On Behalf Of Ramin H.A.
Sent: 23 September 2024 15:46
To: obiba...@googlegroups.com
Subject: Re: [OBiBa] Opal not starting - opal-config?
The good news is that the actual data (datasources, view and identifiers) is not in the OrientDB as this database is primarily for configurations.
If you have time, you could:
·
Backup the current volumes to another folder where you’d start a new Opal instance (use
rsync -avP to preserve ownership)
· Start a fresh new Opal container with the same version as the previous one
· Configure your new Opal database such that it points to the correct Opal Data and Identifiers DBs (doc), e.g.:
# Old OrientDBselect * from Database # Output+----+-----+--------+------------+-------+--------------+------------------+---------------------------------------------------------------------------------------+-------------------+-------------------+|# |@RID |@CLASS |name |usage |defaultStorage|usedForIdentifiers|mongoDbSettings |created |updated |+----+-----+--------+------------+-------+--------------+------------------+---------------------------------------------------------------------------------------+-------------------+-------------------+|0 |#33:0|Database|_identifiers|STORAGE|false |true |{url=mongodb://mongo:27017/opal_ids, username=, password=, properties=, batchSize=100} |2024-08-06 12:02:35|2024-08-06 12:02:35||1 |#34:0|Database|mongodb |STORAGE|true |false |{url=mongodb://mongo:27017/opal_data, username=, password=, properties=, batchSize=100}|2024-08-06 12:02:36|2024-08-06 12:02:36| # New OrientDBconnect plocal:/data/new-volumes/opal/data/orientdb/opal-config admin adminupdate Database merge {mongoDbSettings: {url:'mongodb://mongo:27017/opal_data?authSource=admin', username:'toto', password:'ABCDFED', properties:'', batchSize:100}} where name = 'opal_data'update Database merge {mongoDbSettings: {url:'mongodb://mongo:27017/opal_ids?authSource=admin', username:'toto', password:'JDHEGEE', properties:'', batchSize:100}} where name <> 'opal_data'Sometimes and not often I use this command to point the Database classes to another IP.
· Stop the new container and copy all relevant volumes from your backup above (work, data, fs, etc) except:
o data/orientdb
o data/keystore
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAMpqDgdQ4M87wwvrXB0AEiTB%2B0TkLMwEtiHUkuYie5TP9idEgQ%40mail.gmail.com.
Tom Bishop
Hi,
Just to close the loop on this one, where the OrientDB had become corrupted. Thankfully I was able to restore from a back up. I hope I don’t have to do that again! And I’m glad the back ups worked J
Thanks for your help with the steps to do this.
Tom