migration 4.5.1. to 6

[herr.s...@googlemail.com]

Hi, everyone,

first a Happy New Year to you all!

I am fighting a little bit to update a small repository from an older 4 version to Fedora 6.

The first step described in https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6, that is using the fcrepo-import-export tool in order to generate .ttl files and directories in the file system, works for me, but then I get some Infinispan related errors trying to upgrade to Fedora 5 (before moving on to F6).

So my idea is that first I should ugrade from 4.5.1 to 4.7.5, but how? The instruction given here: https://wiki.lyrasis.org/display/FF/Release+Testing+-+4.7.0#ReleaseTesting4.7.0-MigrationofExistingRepositories leads me to errors.

Any idea, best practice?

Oliver

[David Wilcox]

Hi Oliver,

You are correct that you should upgrade to Fedora 4.7.5 first; the process is documented here: https://wiki.lyrasis.org/display/FF/Fedora+4.7.0+Release+Notes#Fedora4.7.0ReleaseNotes-ModeShape5Upgrade. Basically, you need to:

1. Upgrade to the 4.6.1 patch release: https://wiki.lyrasis.org/display/FF/Fedora+4.6.1+Release+Notes

2. Perform a backup of the 4.6.1 repository: https://wiki.duraspace.org/display/FEDORA47/RESTful+HTTP+API+-+Backup+and+Restore

3. Install Fedora 4.7.5: https://wiki.lyrasis.org/display/FF/Fedora+4.7.5+Release+Notes

4. Perform a restore of the 4.6.1 backup to the 4.7.5 repository

From there you should be able to follow the steps to upgrade from 4.7.5 to 6.0. Let me know if these steps work for you or if you run into any other issues. I’ll update the documentation accordingly so this is clear to others.

Regards,

David

--
David Wilcox
Program Leader, DuraSpace Community Supported Programs
david....@lyrasis.org

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/1c1a96ec-df29-41db-86c3-e79d11152b6cn%40googlegroups.com.

[herr.s...@googlemail.com]

Thank you, David, for this clarification!

Right now I succeeded updating from 4.5.1 to 4.6.1 - I will give you more feedback next week.

CU

Oliver

[herr.s...@googlemail.com]

Hi, David,

I am a little bit further, now working with the correct and latest version of fcrepo-upgrade-utils. But when I am executing

java -jar fcrepo-upgrade-utils-<latest version>.jar \

  -i my-4.7.5-export \

  -o my-5.1.1-export \

  -s 4.7.5 \

  -t 5+

I am told to provide a baseUri. So must there be a Fedora running? And which one? Fedora 4 or Fedora 5?

[David Wilcox]

Hi Oliver,

I checked with Danny Bernstein on this and it looks like this is a bug - that parameter should only be required in the Fedora 5 to 6 upgrade. There is an issue for the bug here: https://jira.lyrasis.org/browse/FCREPO-3599.

In the meantime, you should be able to provide the base URI as a parameter to get the utility to run in its current state. You don’t need to have a running Fedora; you can just use the base URI normally associated with your repository (e.g. http://localhost:8080/rest). 

David

--
David Wilcox
Program Leader, DuraSpace Community Supported Programs
david....@lyrasis.org

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/d0dac6be-6476-4a98-8820-978991afe655n%40googlegroups.com.

[herr.s...@googlemail.com]

Thanks, David (and Danny),

that helped! But now I cannot migrate from 5.1.1 to Fedora 6:

When I  do

java -jar fcrepo-upgrade-utils-<latest-version>.jar \

  -i my-5.1.1-export \

  -o my-fcrepo-6-home \

  -s 5+ \

  -t 6+ \

  -u http://localhost:8080/rest

I get a NullPointerException:

java.lang.NullPointerException: null
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.calculateStateToken(ResourceMigrator.java:516)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.createCommonHeaders(ResourceMigrator.java:458)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.createContainerHeaders(ResourceMigrator.java:470)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrateContainerVersion(ResourceMigrator.java:179)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrateContainer(ResourceMigrator.java:167)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrate(ResourceMigrator.java:126)
        at org.fcrepo.upgrade.utils.f6.MigrateResourceTask.run(MigrateResourceTask.java:59)
        at org.fcrepo.upgrade.utils.f6.MigrationTaskManager.lambda$submit$0(MigrationTaskManager.java:66)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:835)

This structured is created in the target directory

.
└── data
    ├── ocfl-root
    │   ├── 0003-hashed-n-tuple-trees.json
    │   ├── 0=ocfl_1.0
    │   ├── ocfl_1.0.txt
    │   └── ocfl_layout.json
    ├── ocfl-temp
    └── staging

but nothing more. I am using Java 12 and 6.0.0-alpha-2. of the fcrepo-upgrade-utils, no Fedora running. More ideas?

[Daniel Bernstein]

Hi Oliver, 

There appears to be an issue with the last modified date not being reliably non-null in pre-4.7.5 version of Fedora.  I will update fcrepo-upgrade-utils to address the issue. 

You can track progress here: https://jira.lyrasis.org/browse/FCREPO-3607.  I've got a PR for that: https://github.com/fcrepo-exts/fcrepo-upgrade-utils/pull/33

I've built the artifact and attached it to the above JIRA for you to test.

Let me know if you need help.

--Danny Bernstein 

---

From: 'herr.s...@googlemail.com' via Fedora Tech <fedor...@googlegroups.com>
Sent: Tuesday, January 19, 2021 4:14 AM
To: Fedora Tech <fedor...@googlegroups.com>
Subject: Re: [fedora-tech] migration 4.5.1. to 6

 

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/49e7489c-b5e3-4d83-b030-3a35b8f7c931n%40googlegroups.com.

[Daniel Bernstein]

Hi Oliver - 

Here is a link to the compiled version of fcrepo-upgrade-utils to test: https://jira.lyrasis.org/secure/attachment/20424/fcrepo-upgrade-utils-6.0.0-SNAPSHOT.jar in case it wasn't clear from the previous message.

--db

---

From: fedor...@googlegroups.com <fedor...@googlegroups.com> on behalf of Daniel Bernstein <daniel.b...@lyrasis.org>
Sent: Thursday, January 21, 2021 11:43 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/BY5PR22MB176101C4E639D4FFE4E4A48E9AA19%40BY5PR22MB1761.namprd22.prod.outlook.com.

[herr.s...@googlemail.com]

Hi Danny, thanks for your help - unfortunately with the SNAPSHOT jar exactly the same error and stack trace remain.

[Daniel Bernstein]

Hi Oliver,  

It sounds like the create timestamp is not present either.   To verify that, can you drop the *.ttl files in your 5.1.1-export directory associated with the resource that is blowing up during the migration? 

Thanks, 

Danny 

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/3f8701a9-6ce9-4d14-974c-f7c98368b57cn%40googlegroups.com.

[herr.s...@googlemail.com]

Hi Danny,

as far as I can see, every "normal" turtle file has a created and a modified timestamp, but I will verify that by a little script. The rest.ttl file on the other hand, representing the repo's root, does not. Might that be the cause, and would it make sense just inserting dates manually?

See you

Oliver

[Peter Winckles]

Oliver,

 

Would it be possible for you to provide us with a copy of the RDF that is failing migration? It seems to be missing some required properties.

 

Additionally, if you would like more immediate help, we could also help you on Slack (https://wiki.lyrasis.org/display/FF/Mailing+Lists+etc#MailingListsetc-Slack).

 

Thanks,

Peter

David

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/3f8701a9-6ce9-4d14-974c-f7c98368b57cn%40googlegroups.com.

[Peter Winckles]

Oliver,

 

It’s good to hear that it’s just the root that’s missing these fields. We’ll update the tool to handle it, but, yes, you should be able to just insert lastModifiedDate and createdDate into rest.ttl to get it to work.

 

Peter

David

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/148aaf88-f8eb-473a-8bcb-f53e9000bcc2n%40googlegroups.com.

[Daniel Bernstein]

Hi Oliver: 

I've updated the PR linked to https://jira.lyrasis.org/browse/FCREPO-3607. I've also updated the binary attached to the ticket. Would you give it a try with your migration? It would be good to confirm that the update resolves this issue.

[FCREPO-3607] NPE on last modified date when migrating from F5.1.1 to F6 - LYRASIS JIRA Fedora Repository Project; FCREPO-3607; NPE on last modified date when migrating from F5.1.1 to F6 jira.lyrasis.org

Much obliged, 

Danny Bernstein

---

From: 'herr.s...@googlemail.com' via Fedora Tech <fedor...@googlegroups.com>

Sent: Monday, January 25, 2021 12:35 AM

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/148aaf88-f8eb-473a-8bcb-f53e9000bcc2n%40googlegroups.com.

[herr.s...@googlemail.com]

Hi, Danny,

unfortunetaly, the new snapshot leads to a different error:

java.lang.RuntimeException: Failed to migrate resource info:fedora
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.doInSession(ResourceMigrator.java:436)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrateContainerVersion(ResourceMigrator.java:181)

        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrateContainer(ResourceMigrator.java:167)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.migrate(ResourceMigrator.java:126)
        at org.fcrepo.upgrade.utils.f6.MigrateResourceTask.run(MigrateResourceTask.java:59)
        at org.fcrepo.upgrade.utils.f6.MigrationTaskManager.lambda$submit$0(MigrationTaskManager.java:66)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:835)

Caused by: java.lang.NullPointerException: null
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.lambda$migrateContainerVersion$0(ResourceMigrator.java:184)
        at org.fcrepo.upgrade.utils.f6.ResourceMigrator.doInSession(ResourceMigrator.java:433)
        ... 10 common frames omitted

Hope, this is providing you with enough information. But we can chat on Slack or elsewhere, if you want to know more ...

[herr.s...@googlemail.com]

And one additional remark: It seems that the original 4.5.1 repository does not have a fedora:created nor fedora:lastModified date in the root (the rest endpoint). That was no problem until untip the upgrade to 4.7.5, but afterwards the upgrade script seems to need it. Does this explanation make sense to you?

[herr.s...@googlemail.com]

And one more problem: The creation of the OCFL files succeeded, but when I try to start the Fedora 6 repository, I get the following error with fcrepo-webapp-6.0.0-alpha-2-jetty-console.jar:

Exception: Error creating bean with name 'shiroFilter' defined in class path resource [spring/fcrepo-config.xml]: Cannot resolve reference to bean 'securityManager' while setting bean property 'securityManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'securityManager' defined in class path resource [spring/fcrepo-config.xml]: Cannot create inner bean 'util:set#31133b6e' of type [org.springframework.beans.factory.config.SetFactoryBean] while setting bean property 'realms'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'util:set#31133b6e': Cannot resolve reference to bean 'webACAuthorizingRealm' while setting bean property 'sourceSet' with key [0]; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webACAuthorizingRealm': Unsatisfied dependency expressed through field 'rolesProvider'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'webACRolesProvider': Unsatisfied dependency expressed through field 'resourceFactory'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'resourceFactoryImpl': Unsatisfied dependency expressed through field 'persistentStorageSessionManager'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'ocflPersistentSessionManager': Unsatisfied dependency expressed through field 'objectSessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'ocflObjectSessionFactory' defined in org.fcrepo.persistence.ocfl.impl.OcflPersistenceConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.fcrepo.storage.ocfl.OcflObjectSessionFactory]: Factory method 'ocflObjectSessionFactory' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'repository' defined in org.fcrepo.persistence.ocfl.impl.OcflPersistenceConfig: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [edu.wisc.library.ocfl.api.MutableOcflRepository]: Factory method 'repository' threw exception; nested exception is java.lang.IllegalStateException: Failed to find an implementation for storage layout extension 0004-hashed-n-tuple-storage-layout

Without the option -Dfcrepo.home everything is fine, so there might be some problem with the migrated files.

[Peter Winckles]

The problem here is that you used a post alpha-2 version of the migration code with the alpha-2 Fedora. Since the alpha-2 release, the OCFL storage layout extension that Fedora uses[1] was approved, which resulted in some naming changes that the older ocfl-java library is not aware of.

 

One way to resolve this problem, apart from using a more recent version of Fedora, is to delete the oclf_layout.json file that’s in the root of your OCFL repository.

 

Peter

 

[1] https://ocfl.github.io/extensions/0004-hashed-n-tuple-storage-layout.html

David

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/de776fa3-ce06-441e-99ee-b3184fc82fb3n%40googlegroups.com.