INFO - migration utils - FEDORA 4.4 client java API - webapp

[Daniel Mendes da Silva]

Hi,

With the help of Andrew Woods I was able to install FEDORA 4.4 in one of our integration environments.

I had to rebuild the fcrepo4-master project to get the latest changes.

It seems there has been already some bug fixes not available in the released version 4.4.0

 

Currently we have a running version of FEDORA 4.4 but we have no idea:

·         how to migrate the data

·         how the data will be available once migrated

o    will it be accessible using the same identifiers used in FEDORA 3.X.X?

o    will it keep a similar data structure ?

 

This is quite important for us.

Without this we will not be able to go forward with FEDORA 4.4

 

For Fedora 3.8/3.6 we had FedoraClient that was responsible for the creation and execution of all Fedora requests.

Is there something similar for Fedora4.4? I've found  https://github.com/fcrepo4-labs/fcrepo4-client but there are modules missing I could not build it.

Plus we had a database connection with Fedora 3.XX does Fedora 4.4 also needs a db connection? Is this data migrated as well?

Also how come there’s no search engine? In the webapp I see the forms to create/update data but nothing to search like in Fedora 3.XX.

Finally is Fedora 4.4 accepting the same FOXML files for the data ingestion?

Thanks in advance.

[Andrew Woods]

Hello Daniel,

As background information, about three years ago the Fedora community recognized that the 3.x repository software was not positioned to meet the requirements of contemporary use cases. Therefore, a significant amount of community effort has since gone into developing the redesigned 4.x platform.

If it is correct to assume you are not running a Hydra stack, a good starting point for planning your migration are these slides:
https://docs.google.com/presentation/d/1g9f_VMhfD4JDWKtyW6sLuT6N_JO-m2cPQulE1pgM4No/edit?usp=sharing

As mentioned in the slides, there is a "migration-utils" [1] tool that may also be helpful in performing the migration of data.

You may want to become familiar with the REST API [2] for understanding how the migrated data will be available. In order to serve migrated resources from legacy 3.x URLs, a proxying/mapping layer will likely be required.

Regarding the fcrepo4-client [3], that is an "experimental" project that will likely be supplanted by a stateless alternative in the near future. However, it is surprising that you have run into build errors. The following sequence of commands produces a successful build locally:

> rm -rf ~/.m2/repository/org/fcrepo

> git clone https://github.com/fcrepo4-labs/fcrepo4-client.git
> cd fcrepo4-client/
> mvn clean install

Please provide your build output if you would like help debugging the issue.

Regards,

Andrew
[1] https://github.com/fcrepo4-exts/migration-utils
[2] https://wiki.duraspace.org/display/FEDORA44/RESTful+HTTP+API
[3] https://github.com/fcrepo4-labs/fcrepo4-client

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at http://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

[Daniel Mendes da Silva]

So now I managed to have the migration-utils working with the example I wanted.
But now it's throwing an exception whenever I try to migrate our "custom" data


-------------------
-bash-3.00$ java -jar migration-utils-4.4.1-SNAPSHOT-driver.jar conf/fedora3-akubra.xml
INFO 12:35:34.329 (InternalIDResolver) No index directory specified.  Creating temporary index at "/var/tmp/tempfile5358247712058900851basedir/index".
INFO 12:35:34.456 (InternalIDResolver) Builidng an index of all the datastreams in "/opt2/cellar/fedora-commons/data/datastreamStore"...
INFO 12:40:37.146 (Migrator) Processing "changeme:2"...
DEBUG 12:40:37.264 (BasicObjectVersionHandler) Considering object changeme:2 version at 2015-08-14T13:51:38.933Z.
WARN 12:40:37.740 (RDFLanguages) java-jsonld classes not on the classpath - JSON-LD input-output not available.
WARN 12:40:37.740 (RDFLanguages) Minimum jarfiles are jsonld-java, jackson-core, jackson-annotations
WARN 12:40:37.740 (RDFLanguages) If using a Jena distribution, put all jars in the lib/ directory on the classpath
DEBUG 12:40:38.144 (BasicObjectVersionHandler) Considering changed datastream version DC1.0
DEBUG 12:40:38.177 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/identifier value changeme:2
INFO 12:40:39.128 (Migrator) Processing "changeme:1"...
DEBUG 12:40:39.134 (BasicObjectVersionHandler) Considering object changeme:1 version at 2015-07-22T01:04:33.964Z.
DEBUG 12:40:39.350 (BasicObjectVersionHandler) Considering changed datastream version DC1.0
DEBUG 12:40:39.361 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/identifier value changeme:1
INFO 12:40:39.750 (Migrator) Processing "fedora-system:ContentModel-3.0"...
DEBUG 12:40:39.785 (BasicObjectVersionHandler) Considering object fedora-system:ContentModel-3.0 version at 2008-07-02T05:09:44.015Z.
DEBUG 12:40:39.979 (BasicObjectVersionHandler) Considering changed datastream version DC1.0
DEBUG 12:40:39.988 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/identifier value fedora-system:ContentModel-3.0
DEBUG 12:40:39.989 (BasicObjectVersionHandler) Adding http://purl.org/dc/elements/1.1/title value Content Model Object for Content Model Objects
DEBUG 12:40:39.989 (BasicObjectVersionHandler) Considering changed datastream version RELS-EXT1.0
DEBUG 12:40:40.030 (BasicObjectVersionHandler) Considering changed datastream version DS-COMPOSITE-MODEL1.0
DEBUG 12:40:40.637 (BasicObjectVersionHandler) Considering object fedora-system:ContentModel-3.0 version at 2015-05-27T15:29:29.394Z.
DEBUG 12:40:40.638 (BasicObjectVersionHandler) Considering changed datastream version ONTOLOGY1.0
ERROR 12:40:40.741 (FedoraRepositoryImpl) error creating resource http://localhost:9080/fedora/rest/0c/c5/f0/2d/0cc5f02d-ce3a-4ac6-9746-24f5bc0dfd21/ONTOLOGY: 400 Bad Request
ERROR 12:40:40.745 (FedoraRepositoryImpl) could not encode URI parameter
org.fcrepo.client.FedoraException: error retrieving resource http://localhost:9080/fedora/rest/0c/c5/f0/2d/0cc5f02d-ce3a-4ac6-9746-24f5bc0dfd21/ONTOLOGY: 400 Bad Request
        at org.fcrepo.client.impl.FedoraRepositoryImpl.createDatastream(FedoraRepositoryImpl.java:156) ~[migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.f4clients.DefaultFedora4Client.createOrUpdateNonRDFResource(DefaultFedora4Client.java:87) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.handlers.BasicObjectVersionHandler.processObjectVersions(BasicObjectVersionHandler.java:210) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.handlers.VersionAbstractionFedoraObjectHandler.processObject(VersionAbstractionFedoraObjectHandler.java:110) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:89) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.processObject(FoxmlInputStreamFedoraObjectProcessor.java:150) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.Migrator.run(Migrator.java:125) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
        at org.fcrepo.migration.Migrator.main(Migrator.java:61) [migration-utils-4.4.1-SNAPSHOT-driver.jar:na]
Exception in thread "main" java.lang.RuntimeException: org.fcrepo.client.FedoraException: org.fcrepo.client.FedoraException: error retrieving resource http://localhost:9080/fedora/rest/0c/c5/f0/2d/0cc5f02d-ce3a-4ac6-9746-24f5bc0dfd21/ONTOLOGY: 400 Bad Request
        at org.fcrepo.migration.f4clients.DefaultFedora4Client.createOrUpdateNonRDFResource(DefaultFedora4Client.java:90)
        at org.fcrepo.migration.handlers.BasicObjectVersionHandler.processObjectVersions(BasicObjectVersionHandler.java:210)
        at org.fcrepo.migration.handlers.VersionAbstractionFedoraObjectHandler.processObject(VersionAbstractionFedoraObjectHandler.java:110)
        at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:89)
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.processObject(FoxmlInputStreamFedoraObjectProcessor.java:150)
        at org.fcrepo.migration.Migrator.run(Migrator.java:125)
        at org.fcrepo.migration.Migrator.main(Migrator.java:61)
Caused by: org.fcrepo.client.FedoraException: org.fcrepo.client.FedoraException: error retrieving resource http://localhost:9080/fedora/rest/0c/c5/f0/2d/0cc5f02d-ce3a-4ac6-9746-24f5bc0dfd21/ONTOLOGY: 400 Bad Request
        at org.fcrepo.client.impl.FedoraRepositoryImpl.createDatastream(FedoraRepositoryImpl.java:160)
        at org.fcrepo.migration.f4clients.DefaultFedora4Client.createOrUpdateNonRDFResource(DefaultFedora4Client.java:87)
        ... 6 more
Caused by: org.fcrepo.client.FedoraException: error retrieving resource http://localhost:9080/fedora/rest/0c/c5/f0/2d/0cc5f02d-ce3a-4ac6-9746-24f5bc0dfd21/ONTOLOGY: 400 Bad Request
        at org.fcrepo.client.impl.FedoraRepositoryImpl.createDatastream(FedoraRepositoryImpl.java:156)
        ... 7 more
INFO 12:40:40.748 (InternalIDResolver) Deleting index directory at "/var/tmp/tempfile5358247712058900851basedir/index"...

[Daniel Mendes da Silva]

Hello,
thank you for the slides presentation, its quite informative.

I still have issues using the tool.
Currently I have a running Fedora 4.4 instance and each time I try to run the migration-utils with an example I have the following stack trace:

--------------------------------------------------------------------------------------------------------------------------------------------------------
-bash-3.00$ pwd
/opt2/cellar/FEDORA/migration-utils-zip/migration-utils
-bash-3.00$ ls -rtlah
total 40518
-rw-rw-r--   1 cellar   gcellar     7.0K Oct  8 20:54 README.md
drwxrwxr-x   2 cellar   gcellar        8 Oct 15 16:21 conf
drwxrwxr-x   3 cellar   gcellar        3 Oct 15 16:24 src
-rw-rw-r--   1 cellar   gcellar      20M Nov  2 13:48 migration-utils-4.4.0-driver.jar
drwxrwxr-x   3 cellar   gcellar        3 Nov  3 17:14 ..
drwxrwxr-x   2 cellar   gcellar        3 Nov  3 17:17 backup
drwxrwxr-x   5 cellar   gcellar        7 Nov 10 09:36 .
-bash-3.00$ java -jar migration-utils-4.4.0-driver.jar conf/fedora3-akubra.xml
INFO 09:37:52.803 (InternalIDResolver) No index directory specified.  Creating temporary index at "/var/tmp/tempfile4831344035005288418basedir/index".
INFO 09:37:52.944 (InternalIDResolver) Builidng an index of all the datastreams in "src/test/resources/akubraFS/datastreamStore"...
INFO 09:37:53.421 (Migrator) Processing "example:1"...
DEBUG 09:37:53.583 (BasicObjectVersionHandler) Considering object example:1 version at 2015-01-27T19:07:33.120Z.
Exception in thread "main" java.lang.NoSuchFieldError: langString
        at com.hp.hpl.jena.sparql.graph.NodeConst.<clinit>(NodeConst.java:46)
        at com.hp.hpl.jena.sparql.engine.optimizer.reorder.ReorderFixed.<clinit>(ReorderFixed.java:59)
        at com.hp.hpl.jena.sparql.engine.optimizer.reorder.ReorderLib.fixed(ReorderLib.java:71)
        at com.hp.hpl.jena.sparql.engine.main.StageGeneratorGeneric.<clinit>(StageGeneratorGeneric.java:41)
        at com.hp.hpl.jena.sparql.engine.main.StageBuilder.standardGenerator(StageBuilder.java:108)
        at com.hp.hpl.jena.sparql.engine.main.StageBuilder.init(StageBuilder.java:74)
        at com.hp.hpl.jena.query.ARQ.init(ARQ.java:532)
        at com.hp.hpl.jena.query.ARQ.<clinit>(ARQ.java:547)
        at org.apache.jena.riot.system.RiotLib.<clinit>(RiotLib.java:63)
        at org.apache.jena.riot.lang.LangRDFXML.<init>(LangRDFXML.java:105)
        at org.apache.jena.riot.lang.LangRDFXML.create(LangRDFXML.java:76)
        at org.apache.jena.riot.RiotReader.createParser(RiotReader.java:134)
        at org.apache.jena.riot.RiotReader.parse(RiotReader.java:118)
        at org.fcrepo.client.utils.HttpHelper.loadProperties(HttpHelper.java:343)
        at org.fcrepo.client.impl.FedoraRepositoryImpl.getObject(FedoraRepositoryImpl.java:133)
        at org.fcrepo.client.impl.FedoraRepositoryImpl.createResource(FedoraRepositoryImpl.java:235)
        at org.fcrepo.migration.f4clients.DefaultFedora4Client.createPlaceholder(DefaultFedora4Client.java:126)
        at org.fcrepo.migration.idmappers.OpaqueIDMapper.mapObjectPath(OpaqueIDMapper.java:136)
        at org.fcrepo.migration.handlers.BasicObjectVersionHandler.processObjectVersions(BasicObjectVersionHandler.java:181)

        at org.fcrepo.migration.handlers.VersionAbstractionFedoraObjectHandler.processObject(VersionAbstractionFedoraObjectHandler.java:110)
        at org.fcrepo.migration.handlers.ObjectAbstractionStreamingFedoraObjectHandler.completeObject(ObjectAbstractionStreamingFedoraObjectHandler.java:89)
        at org.fcrepo.migration.foxml.FoxmlInputStreamFedoraObjectProcessor.processObject(FoxmlInputStreamFedoraObjectProcessor.java:150)
        at org.fcrepo.migration.Migrator.run(Migrator.java:125)
        at org.fcrepo.migration.Migrator.main(Migrator.java:61)

INFO 09:37:54.164 (InternalIDResolver) Deleting index directory at "/var/tmp/tempfile4831344035005288418basedir/index"...
-bash-3.00$ java -version
java version "1.8.0_65"
Java(TM) SE Runtime Environment (build 1.8.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
--------------------------------------------------------------------------------------------------------------------------------------------------------

[Daniel Mendes da Silva]

In order to assure that I was building correctly the migration-utils jar I've downloaded the parent project and added fcrepo4-client and migration-utils as child modules
The migration-utils makes reference to the parent and the fcrepo4-client. Otherwise I would have to change the pom - this way I'm using the same as you are.
The build was successful.

And :) the migration utils as well. At least the example worked without error.
I will try to continue
Next step is to see how I can control how the data is migrated and how to access it

---------------------------------------- mvn clean package result ----------------------------------------
[INFO] Fedora Commons 4 :: Parent POM .................... SUCCESS [0.202s]
[INFO] Fedora Commons 4 .................................. SUCCESS [0.217s]
[INFO] Fedora Repository BOM ............................. SUCCESS [0.008s]
[INFO] Fedora Repository JCR BOM ......................... SUCCESS [0.003s]
[INFO] Fedora Repository BOMs ............................ SUCCESS [0.003s]
[INFO] Fedora Repository Kernel API ...................... SUCCESS [0.311s]
[INFO] Fedora Repository Metrics Module .................. SUCCESS [0.080s]
[INFO] Fedora Repository Minter Module ................... SUCCESS [0.100s]
[INFO] Fedora Repository Configurations Module ........... SUCCESS [0.030s]
[INFO] Fedora Repository Kernel Implementation (ModeShape)  SUCCESS [0.540s]
[INFO] Fedora Repository JMS Module ...................... SUCCESS [0.110s]
[INFO] Fedora Repository Object Serialization Module ..... SUCCESS [0.080s]
[INFO] Fedora Repository HTTP Commons Module ............. SUCCESS [0.424s]
[INFO] Fedora Repository FileSystem Connector Module ..... SUCCESS [0.120s]
[INFO] Fedora Repository Authorization Commons Module .... SUCCESS [0.120s]
[INFO] Fedora Repository HTTP API Module ................. SUCCESS [0.220s]
[INFO] Fedora Repository Deployable Web Application ...... SUCCESS [0.320s]
[INFO] Fedora Repository LDP integration tests Module .... SUCCESS [0.040s]
[INFO] Fedora Repository RDF integration tests Module .... SUCCESS [0.060s]
[INFO] fcrepo4-client .................................... SUCCESS [0.010s]
[INFO] Fedora Repository Java Client API ................. SUCCESS [0.080s]
[INFO] Fedora Repository Java Client API Impl ............ SUCCESS [0.110s]
[INFO] migration-utils ................................... SUCCESS [0.282s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.944s
[INFO] Finished at: Tue Nov 10 10:47:30 CET 2015
[INFO] Final Memory: 17M/229M
[INFO] ------------------------------------------------------------------------

On Friday, November 6, 2015 at 3:20:54 PM UTC+1, Daniel Mendes da Silva wrote:

Hi,

With the help of Andrew Woods I was able to install FEDORA 4.4 in one of our integration environments.

I had to rebuild the fcrepo4-master project to get the latest changes.

It seems there has been already some bug fixes not available in the released version 4.4.0

 

Currently we have a running version of FEDORA 4.4 but we have no idea:

·         how to migrate the data

·         how the data will be available once migrated

o    will it be accessible using the same identifiers used in FEDORA 3.X.X?

o    will it keep a similar data structure ?

 

This is quite important for us.

Without this we will not be able to go forward with FEDORA 4.4

 

...

[Jeevan Patnaik]

Hi, 

I am getting the same issue of encoding URI parameter. Could you please let me know how it got resolved?

Regards,

Jeevan.