Import Issue

[clsamc...@gmail.com]

Greetings,

I have launched a new instance of Opal to replace a machine being retired.

I am finding that on some tables, I am unable to import. Both Opals are 4.7.3.

The error is as follows, below. Guidance would be appreciated. Other tables import successfully.

Thanks.

2024-08-20 10:13:54,354 [pool-14-thread-1] INFO  org.obiba.opal.rest.client.magma.OpalJavaClient - Disconnecting from Opal: https://opal.clsa-elcv.ca:8843/ws/
2024-08-20 10:13:54,370 [pool-14-thread-1] ERROR org.obiba.opal.shell.commands.ImportCommand - Runtime error while importing data
java.lang.RuntimeException: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.bson.Document
at org.obiba.opal.core.service.DataImportServiceImpl.copyValueTables(DataImportServiceImpl.java:137)
at org.obiba.opal.core.service.DataImportServiceImpl.importData(DataImportServiceImpl.java:103)
at org.obiba.opal.core.service.DataImportServiceImpl.importData(DataImportServiceImpl.java:83)
at org.obiba.opal.shell.commands.ImportCommand.importFromTables(ImportCommand.java:166)
at org.obiba.opal.shell.commands.ImportCommand.executeImports(ImportCommand.java:100)
at org.obiba.opal.shell.commands.ImportCommand.execute(ImportCommand.java:77)
at org.obiba.opal.shell.CommandJob.run(CommandJob.java:156)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast to org.bson.Document
at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.removeFile(MongoDBValueTableWriter.java:182)
at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.updateBinary(MongoDBValueTableWriter.java:176)
at org.obiba.magma.datasource.mongodb.MongoDBValueTableWriter$MongoDBValueSetWriter.writeValue(MongoDBValueTableWriter.java:152)
at org.obiba.magma.support.DatasourceCopier.copyValues(DatasourceCopier.java:304)
at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.copyValue(MultithreadedDatasourceCopier.java:371)
at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.run(MultithreadedDatasourceCopier.java:360)
at org.obiba.magma.support.MultithreadedDatasourceCopier.write(MultithreadedDatasourceCopier.java:195)
at org.obiba.magma.support.MultithreadedDatasourceCopier.copy(MultithreadedDatasourceCopier.java:182)
at org.obiba.opal.core.service.CopyValueTablesLockingAction$CopyAction.execute(CopyValueTablesLockingAction.java:130)
at org.obiba.opal.core.magma.concurrent.LockingActionTemplate$1.doInTransactionWithoutResult(LockingActionTemplate.java:47)
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.doInTransaction(LockingActionTemplate.java:42)
at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.execute(LockingActionTemplate.java:31)
at org.obiba.opal.core.service.DataImportServiceImpl.copyValueTables(DataImportServiceImpl.java:129)
... 13 common frames omitted
2024-08-20 10:13:54,469 [pool-14-thread-1] WARN  org.obiba.opal.shell.commands.ImportCommand - Import failed in 33.91 s

[Yannick Marcon]

Hi,

Does your table contain 'binary' type variables? 

Are they repeatable?

What is the version oof the Mongodb database?

Regards

Yannick

--
You received this message because you are subscribed to the Google Groups "obiba-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to obiba-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/ef5ae623-c6b6-4b8b-84ae-9e7124779270n%40googlegroups.com.

[CLSA IT]

Hi Yannick,

They do contain binary data. I'm not sure what is meant by repeatable.

On the server with the data, the Mongo version is 7.0.8 and on the server performing the import the version is 7.0.2.

I should note, only three tables out of 86 are failing.

Thank you,

-- 
Sean Hurley

You received this message because you are subscribed to a topic in the Google Groups "obiba-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/obiba-users/u7NW4GaNF2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to obiba-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAGrE6osOM%2BfJNiyVxNwcyMNM-WLpQBx%2B_2gMzt9a28YaO6r7rw%40mail.gmail.com.

[clsamc...@gmail.com]

I'm following up. Thanks.

[Elson, Dwain]

Hi Yannick,

Have you been able find out anything about this issue?

Regards,

Dwain

---

From: obiba...@googlegroups.com <obiba...@googlegroups.com> on behalf of clsamc...@gmail.com <clsamc...@gmail.com>
Sent: Monday, August 26, 2024 10:06 AM
To: obiba-users <obiba...@googlegroups.com>
Subject: Re: [OBiBa] Import Issue

 

Caution: External email.

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/893ec3c2-6729-4e0f-8e94-4fc453a51c72n%40googlegroups.com.

[Yannick Marcon]

Hi Dwain,

I will try to reproduce that issue tomorrow.

Regards,

Yannick

[clsamc...@gmail.com]

Hi Yannick

Working with this, we tried to export one of the affected tables. We are unable to. It quits pretty quickly and provides the following error:

java.lang.RuntimeException: org.obiba.magma.MagmaRuntimeException: Cannot retrieve content of gridFsFile [636b27bc808de225bb577a0e]
at org.obiba.opal.core.service.DataExportServiceImpl.exportTablesToDatasource(DataExportServiceImpl.java:88)
at org.obiba.opal.shell.commands.CopyCommand.execute(CopyCommand.java:168)

at org.obiba.opal.shell.CommandJob.run(CommandJob.java:156)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Caused by: org.obiba.magma.MagmaRuntimeException: Cannot retrieve content of gridFsFile [636b27bc808de225bb577a0e]
at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getByteArray(MongoDBValueLoaderFactory.java:84)
at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getValue(MongoDBValueLoaderFactory.java:71)
at org.obiba.magma.Value.getValue(Value.java:60)
at org.obiba.magma.type.AbstractValueType.toString(AbstractValueType.java:130)
at org.obiba.magma.Value.toString(Value.java:104)
at org.obiba.magma.xstream.converter.ValueConverter.marshal(ValueConverter.java:43)
at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
at org.obiba.magma.xstream.converter.ValueSequenceConverter.marshal(ValueSequenceConverter.java:51)
at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:270)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:174)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:262)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:270)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:174)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:258)
at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
at com.thoughtworks.xstream.core.TreeMarshaller.convert(TreeMarshaller.java:71)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1303)
at com.thoughtworks.xstream.XStream.marshal(XStream.java:1292)
at com.thoughtworks.xstream.XStream.toXML(XStream.java:1265)
at org.obiba.magma.datasource.fs.FsValueTableWriter$XStreamValueSetWriter.close(FsValueTableWriter.java:109)
at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.copyValue(MultithreadedDatasourceCopier.java:372)

at org.obiba.magma.support.MultithreadedDatasourceCopier$ConcurrentValueSetWriter.run(MultithreadedDatasourceCopier.java:360)
at org.obiba.magma.support.MultithreadedDatasourceCopier.write(MultithreadedDatasourceCopier.java:195)
at org.obiba.magma.support.MultithreadedDatasourceCopier.copy(MultithreadedDatasourceCopier.java:182)

at org.obiba.opal.core.service.DataExportServiceImpl$ExportActionTemplate$ExportAction.exportTableToDatasource(DataExportServiceImpl.java:197)
at org.obiba.opal.core.service.DataExportServiceImpl$ExportActionTemplate$ExportAction.execute(DataExportServiceImpl.java:166)

at org.obiba.opal.core.magma.concurrent.LockingActionTemplate$1.doInTransactionWithoutResult(LockingActionTemplate.java:47)
at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:36)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.doInTransaction(LockingActionTemplate.java:42)
at org.obiba.opal.core.magma.concurrent.LockingActionTemplate.execute(LockingActionTemplate.java:31)

at org.obiba.opal.core.service.DataExportServiceImpl.exportTablesToDatasource(DataExportServiceImpl.java:80)
... 9 more
Caused by: org.bson.BsonInvalidOperationException: Value expected to be of type STRING is of unexpected type NULL
at org.bson.BsonValue.throwIfInvalidType(BsonValue.java:419)
at org.bson.BsonValue.asString(BsonValue.java:69)
at com.mongodb.client.gridfs.codecs.GridFSFileCodec.decode(GridFSFileCodec.java:66)
at com.mongodb.client.gridfs.codecs.GridFSFileCodec.decode(GridFSFileCodec.java:47)
at com.mongodb.internal.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60)
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
at org.bson.internal.LazyCodec.decode(LazyCodec.java:48)
at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:104)
at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63)
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:51)
at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:535)
at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:420)
at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:337)
at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:116)
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:643)
at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:71)
at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:240)
at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:226)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:126)
at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:116)
at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:345)
at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:232)
at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:695)
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$2(OperationHelper.java:566)
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591)
at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$3(OperationHelper.java:565)
at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:591)
at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:564)
at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:690)
at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:65)
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:722)
at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:86)
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:191)
at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:213)
at com.mongodb.client.gridfs.GridFSBucketImpl.getFileInfoById(GridFSBucketImpl.java:547)
at com.mongodb.client.gridfs.GridFSBucketImpl.openDownloadStream(GridFSBucketImpl.java:260)
at com.mongodb.client.gridfs.GridFSBucketImpl.openDownloadStream(GridFSBucketImpl.java:255)
at org.obiba.magma.datasource.mongodb.MongoDBValueLoaderFactory$MongoDBBinaryValueLoader.getByteArray(MongoDBValueLoaderFactory.java:81)
... 51 more

Thanks.

[Ramin H.A.]

Hi Dwain,

How are you importing from one Opal to another? Did you try to export from source Opal/Project in Opal Archive (ZIP) format and import the archive into the new one? 

Ramin

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/c78fd49f-eae9-4b27-91c5-b46456ecebe0n%40googlegroups.com.

[clsamc...@gmail.com]

Hi Ramin, it will not export in Opal Archive format at all.

Sean.

[clsamc...@gmail.com]

I think it may be related to the MongoDB version. I am able to export using Opal Archive on an older version (3.6.9) of Mongo. The issue appears to be with a newer version, 7.0.8.

[Ramin H.A.]

I suggest you check this documentation of Mongo and see if you need to enable the version feature prior to upgrading your Mongo. For one of our partners I had to do this 5 times to be able to upgrade several versions without any data loss.

Do you guys have a Docker stack for your Opal installation, it would have been so much easier that way. 

FYI: At maelstrom and at some of our partners, the OBiBa stack is configured as Docker containers using a docker compose  file with proper ports and volumes setup. This setup makes the migration and backup so much easier.

In case you consider this setup in the future: https://github.com/obiba/obiba-demo

Best,

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/a914d522-c222-422c-b869-6e226662b1c6n%40googlegroups.com.

[CLSA IT]

The process we followed was to dump the mongo databases into BSON files and then restore them into the new database. I'm not sure where this leaves us.

-- 
Sean Hurley
Systems Administrator
Canadian Longitudinal Study on Aging (CLSA)
Étude longitudinale canadienne sur le vieillissement (ÉLCV)

Mobile/Text: 289-700-1606
Email: shu...@mcmaster.ca

McMaster University
McMaster Innovation Park
175 Longwood Rd. S. Suite 309A
Hamilton, ON L8P 0A1

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAMpqDgefbo%3D4gwbCWAjuzPROqLur586_W34dJ%2BzNCyd3zH%2BuQQ%40mail.gmail.com.

[Ramin H.A.]

I suspect that you might have had to upgrade the host mongo using the link above before. Anyways, the Opal Archive can be an alternative for the three missing ones.

R

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAJCg9M7M0s6qGkYJdDLKVgEXN3T569nbWZGE1g0iaeTWFL8diw%40mail.gmail.com.

[Yannick Marcon]

Hi,

According to the MongoDB compatibility matrix, Opal 4.7.3 would support MongoDB up to version 6.0 (latest release of Opal includes MongoDB java driver 4.7.1).

I have just upgraded the Mongodb driver to the latest version available: please give Opal 4.7.4 a try.

Regards,

Yannick

To view this discussion on the web visit https://groups.google.com/d/msgid/obiba-users/CAJCg9M4OCvMAKufsnnCh_FR%3D6Od4rt%2BbTJ0Cc6kAow__jLHO-g%40mail.gmail.com.

[clsamc...@gmail.com]

Thanks for the effort, Yannick. Unfortunately. no joy for us:

org.obiba.magma.MagmaRuntimeException: Cannot retrieve content of gridFsFile [623111cf808de21b427dde4e]

Sean.

[clsamc...@gmail.com]

An interesting detail is that this issue does not appear to impact older data.

[rhaeri]

Hi Sean, 

You should consider creating your OBiBa stack using Docker containers, it makes this sort of situations much simpler. For one of our servers I did just that, we had Debian packages of Opal, Mongo, Elasticsearch, and Rock. 

I made sure that my image versions were exactly those of the installed Debian packages and that the docker volumes were pointing to the migrated (rsync-ed) HOME folders. There were some tweaking in the config files so they use the proper service names rather than URLs. The Opal config DB (OrientDB) also had to be edited using OrientDB console so the MongoDB urls were using service names rather than URLs.

Anyways, something to consider in the future.

Best,