Archivematica & Islandora integration

[Rachel Trent]

We were wondering if the Artefactual team might have any upcoming projects for the integration of Islandora and Archivematica? If so, we'd be very interested to hear any information about timelines, specs, & planned workflows (e.g., whether materials are first processed into Archivematica or Islandora, how metadata is handled between the two systems). 

Thanks, 

Rachel 

State Archives of NC

[Rick Sarvas]

We experimented with that idea at the University of Connecticut last year when we were looking for a standardized ingest package format for the bulk Fedora Commons system we are still developing that produces Islandora compatible Fedora objects and data streams as an end product. Our plan was to take the output of Archivematica and use it as the input of our ingest process by extending the Archivematica code a bit. We managed to get some proof of concept code working for processing AIPs produced by Archivematica 0.6, then 0.7 (with some modification) but found that the code no longer worked with later versions. Given that Archivematica was still under aggressive development that time, we decided to wait until Archivematica achieved at least a 1.x status before visiting this idea again.

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Aaron Collie]

We (MSU Libraries) did the same thing, Rick. We wrote code for .7 and it is now obsolete for .9

We are still at it (output of archivematica to islandora ingest) this time we are looking at Tuque or eulfedora. I would be really interested in working together...

Let's see what interest this conversation generates?

-Aaron

[Tim Hutchinson]

Hello,

My institution is currently in active discussion with Artefactual relating to a proposed integration with Islandora - we are at the stage of trying to put together funding.

The proposed integration goes in one direction: ingest of the digital objects into Islandora and entry of metadata in Islandora; and then automated production of the AIP in Archivematica. The AIPs would be stored in the Archivematica AIPstore. It sounds like this is the opposite direction to the development outlined by Aaron and Rick.

If anyone is in a position to discuss potential cost-sharing of this project, I'd be pleased to hear from you off-list.

Tim

On 20/06/2013 10:58 AM, Aaron Collie wrote:

We (MSU Libraries) did the same thing, Rick. We wrote code for .7 and it is now obsolete for .9

We are still at it (output of archivematica to islandora ingest) this time we are looking at Tuque or eulfedora. I would be really interested in working together...

Let's see what interest this conversation generates?

-Aaron

On Thu, Jun 20, 2013 at 10:46 AM, Rick Sarvas <rick....@gmail.com> wrote:

We experimented with that idea at the University of Connecticut last year when we were looking for a standardized ingest package format for the bulk Fedora Commons system we are still developing that produces Islandora compatible Fedora objects and data streams as an end product. Our plan was to take the output of Archivematica and use it as the input of our ingest process by extending the Archivematica code a bit. We managed to get some proof of concept code working for processing AIPs produced by Archivematica 0.6, then 0.7 (with some modification) but found that the code no longer worked with later versions. Given that Archivematica was still under aggressive development that time, we decided to wait until Archivematica achieved at least a 1.x status before visiting this idea again.

On Tue, Jun 18, 2013 at 10:18 AM, Rachel Trent <rachel.tr...@gmail.com> wrote:

We were wondering if the Artefactual team might have any upcoming projects for the integration of Islandora and Archivematica? If so, we'd be very interested to hear any information about timelines, specs, & planned workflows (e.g., whether materials are first processed into Archivematica or Islandora, how metadata is handled between the two systems). 

Thanks, 

Rachel 

State Archives of NC

-- 
Tim Hutchinson
University of Saskatchewan Archives
301 Murray Library, 3 Campus Dr.
Saskatoon, SK S7N 5A4
(306) 966-6028
e-mail: tim.hut...@usask.ca

[Rachel Trent]

It's great to hear that others are actively looking into this as well. Thanks for your responses, Rick, Aaron, and Tim. 

We'd heard some discussion from the 2012 Islandora Camp regarding this very dilemma: whether to process SIPs first into Archivematica or Islandora, or another variation that brings objects out of one tool and then back into it. What initially seemed to make sense to our institution was the type of workflow Aaron and Rick describe, where materials are first processed in Archivematica and the output fed to Islandora, but the opposite seems to make more sense for many other institutions. 

I'm interested to hear if any institutions might be comfortable sharing existing documentation of intended integrated workflows. Or, if your institution considered multiple workflows and documented your decision to choose one over others, we'd be very interested in chatting about your reasoning.

Rachel 

[Justin Simpson]

Artefactual is currently doing development work for the 1.0 release, scheduled for September 2013, that is relevant to this thread.

One of the major features of the 1.0 release will be a new Storage Service.  We are developing a separate component that exposes a REST api, that can be used to manage storage requirements for several Archivematica pipelines.  Individual Archivematica installations will push AIP's to the Storage Service, which will take care of actually storing them. 

In Archivematica 1.0, the Storage Service will work with local file systems, NFS and CIFS (samba).  After the 1.0 release, Artefactual will be working on plugins for the Storage Service that allow Archivematica to store AIP's in LOCKSS and as Hydra objects.  There will most likely be work done on a DuraCloud plugin as well.  The Hydra plugin should be very similar to any Fedora based repository.  The Storage Service would create a single Hydra Object, and attach multiple datastreams to it, for example a preservation copy of a digital object from an AIP would be one stream, and a corresponding access copy from a DIP generated by Archivematica would be another stream.  Archivematica will see the AIP and the DIP as separate packages, the Storage Service will abstract away the fact that both files are stored in the same Hydra object.  The same approach would work for Fedora/Islandora.

These workflows all involve processing SIP's in Archivematica first, and storing an AIP (and possibly DIPs) outside of Archivematica.  The Storage Service will also manage Transfer Source locations, which allows the possibility of alternate workflows.  For example, objects are first processed in another system, like Islandora (allowing metadata entry, possibly generation of derivatives like access copies, etc) , and sent to Archivematica to create an AIP afterwards. Both directions should be possible.

We will be publishing the final Storage Service REST api over this summer; it needs to be finalized before the 1.0 release, so that Archivematica 1.0 can use it.  Anyone interested in working on Archivematica - Islandora integration should probably plan to code against this new api.  Earlier versions of Archivematica have been under fairly intense development change, as a beta product, but with this Storage Service api there will finally be a stable endpoint to code against.

Like Rachel, I am also very interested to hear how different institutions are thinking about integrated workflows. 

Regards,



Justin Simpson
Archivematica System Architect
www.artefactual.com
604-527-2056

Justin Simpson
Archivematica System Architect
www.artefactual.com
604-527-2056

--

[Aaron Collie]

This is all great to hear, especially about the API. In our case we would like the AIP to be left to the filesystem (e.g. bag + backup); with monitoring/preservation services built up from there. We like that, similar to Fedora Commons, AIP indexes can be rebuilt from the contents of the filesystem. That means portability, flexibility, etc. In our case the AIP would then be Externally Referenced (because we anticipate massive preservation masters) by our Fedora Digital Objects, which would increase programatic access to the spaghetti strings that are collections. 

We would like our DIP to be ingest and published via the Islandora Solution Packs, so as to leverage any microservice designed to be triggered by ingest and to leverage viewers designed to display the content.

Our original project was to simply take the METS file created by Archivematica and transform that into the extension of METS used for ingest into Fedora Commons. We did that and presented on it [1] but it was really just a conceptual project.

Now we see the possibility of using Tuque [2] or other tools to ingest items directly to Islandora; so we are moving away from the METS/batch method. We can already send items from Archivematica to FedCom automatically using eulfedora, but this doesn't trigger Islandora stuff.

I see it like this: Archivematica for ingest of SIP and monitoring of AIP, Fedora Commons for middleware registry of spaghetti datastreams; Islandora for display of DIP (e.g. solr, solution packs).

What I think would be interesting is integration of messaging between all these stacks. So that when a new version of a FC datastream is uploaded to a digital object the change is recorded as a premis event... etc

This is really a lot to talk about over email. We should all sit and chat at Some Upcoming Event.

[1] http://connect.ala.org/files/48943/ALAMIG_Mak_Collie_01092013.pptx

[2] https://github.com/Islandora/tuque