Fedora 4 to 5 migration using fcrepo-import-export

[Mohamed Mohideen Abdul Rasheed]

Hello everyone,

At UMD, we are looking at the Fedora 4 to Fedora 5 migration using the fcrepo-import-export tool. 

Here are some of my finding after initial investigation:

1. Import into fcrepo5 mostly fails with 409 and 412 errors

After exporting a small dataset from fcrepo4, I removed all acl:accessControl, ldp:Resource, ldp:Container, ldp:contains triples from the exported ttl files, but the import failed with 409/412 errors. After enabling DEBUG logs on the repository, there were stacktraces with "predicate may not take an object in the server managed namespaces (http://www.w3.org/ns/ldp#)" even though none of the ttl files had  "ldp:Container". See attached file for full stacktace.

2. Binary files are imported but files are 0 bytes.

3. Import/export operations should support retry and resume options.

When I tried exporting a larger container (rdfType: n-triples), the export failed due to a spurious error after about 4 hours. A retry option with a backoff timer could help prevent spurious errors from leading to failure. Also, in our dev instance, the export rate seemed to be about 25K items/hour. At that rate, it would take about a week to finish the export of our current production data. So, we would need to have a way to track progress to be able to resume the operation in case of failure.

More information:

• I tried export / import using both 0.3.0 and 0.4.0-SNAPSHOT versions.
• I enabled "relaxed" mode on the repository before attempting imports.
• I tried fcrepo4 export containing a just couple of simple no-relationship items was successfully imported into fcrepo5.
• The fcrepo4 sample dataset export that was failing to import to fcrepo5 did successfully import to a fcrepo4 instance, but it took 2 passes (dependent items were imported in the secondd pass).

Has anyone encountered similar errors during imports into fcrepo5? Also, If anyone else has experimented with either import-export or other options to migrate from 4-to-5, please share your thoughts.

Thanks

Mohamed Abdul Rasheed

Software Systems Development & Research

B0112 McKeldin Library

University of Maryland, College Park

College Park 20742 

Work: 301-314-2720

[Daniel Bernstein]

Hello Mohammed, 

Thank you for communicating your results.   I'm not surprised that importing into F5 from an F4 export did not work as there are several API changes that have not yet been incorporated in the migration tool.   That said  I have begun working on round-tripping F5.  There are three significant API updates that need to be accounted for.  1) Versioning, 2) ACLs, 3) External content handling, and 4) miscellaneous header changes.

So far we are able to export F5 versions.   I am working on exporting external content, and ACLs.   If it would be helpful I could shift gears and focus on your use case in order to get things going quickly for you.  Regarding #1 below,  I believe that issue can be resolved with a simple Prefer header update.  Regarding #2,  I'm not sure what the issue is but it is likely a straightforward fix as well.  Once I have those in place, it would be very helpful to have your help in testing on your dataset.  

I think #3 is an excellent idea.   Would you mind creating a JIRA  detailing what you would like to see here? https://jira.duraspace.org/secure/Dashboard.jspa

Also if you have time, I'm looking for volunteers to complete the round-tripping of data for F5, both for testing and development.  Please join us on the tech call tomorrow at 11 am EST if you're available so we can discuss next steps and possible collaboration: https://wiki.duraspace.org/display/FF/2019-05-16+-+Fedora+Tech+Meeting. 

Best regards, 

Danny

************************************

Daniel Bernstein

Tech Lead, Duraspace

415.533.2354 (cell)

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/CAA65_OdJXw1UDwTWUoxsdZ2iOqgL6aPJ%2BkQCqUGMti1Sd1pA5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.