BagIt+ORE: What's the current thinking?

[Chris Wilper]

I know a few folks have attempted to combine BagIt + ORE in the past.
Anyone know of someone who has done this recently, or have a system in
operation today that does this? I'm curious how people ultimately
handled the "Protocol-based URI" requirement and pointed to the
content in the bag.

- Chris

[Liz.Sch]

J. McDonough described in a 2009 post about the ORE Protocol-based URI
requirement and Preserving Virtual World's approach to this problem.
He describes using a fetch.txt file to record the URI for file as they
are referenced in the BagIt Packages.

http://groups.google.com/group/digital-curation/browse_thread/thread/f0d8b03b0f9f5d55/8ecee19f3ce6c299?lnk=gst&q=BagIt#8ecee19f3ce6c299

I know DataOne also has some documentation about ORE-BagIt Data
Packaging at http://mule1.dataone.org/ArchitectureDocs-current/design/DataPackage.html

I am also interested in reading about other attempts at BagIt+ORE
combinations, so keep posting as you come across them.

Liz Schlagel
NSIDC

[Matt Jones]

Chris and Liz -- 

Like Liz said, we are planning on using ORE and BagIt together in DataONE.  The document she linked to needs some editing, but is basically correct.  Our working model is that ORE describes the package and gets included in the bag, that the protocol-based URIs for objects in ORE will utilize the URI from an identifier resolver service, but that we will include a dcterms:identifier field for each object in the package in the RDF, which will allow us to distinguish the identifier for the object from the protocol-based URI for the object.  Its that identifier which we expect to use in our naming convention for files in our BagIt bag.  So, the filename will link to the ID, and the ID is associated with the protocol URI in the RDF via dcterms:identifier.  See http://mule1.dataone.org/ArchitectureDocs-current/design/DataPackage.html for an example RDF document showing how we'll relate these.

An alternate that we have considered is to include a mapping from identifier to filename in one of the manifests. But it was hard to choose which one was most appropriate.  Candidates would be a bagit tag file, the ORE document itself, or another manifest file of our design.  None of these seemed like good options to us for various reasons.

For us, what's most important is getting agreement among repositories about how to link and serialize these composite objects.  We would welcome feedback, as well as pointers to other ways groups have done similar models of aggregation and serialization.  

Matt

--
You received this message because you are subscribed to the Google Groups "Digital Curation" group.
To post to this group, send email to digital-...@googlegroups.com.
To unsubscribe from this group, send email to digital-curati...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/digital-curation?hl=en.

[Chris Wilper]

Hi Liz and Matt,

Thanks for the pointers. It confirms my suspicion that people are
still interested in doing a BagIt + ORE kind of thing, but a best
practice hasn't emerged yet.

I also think having some general agreement on how to do this kind of
thing, whether a "hack" or not, could get some good interop. going
among a quite a variety of repository-like things.

Here's a writeup about, and a link to a very relevant Internet-Draft I
finally decided was ready to publish for the first time this morning:

http://cwilper.blogspot.com/2012/04/new-internet-draft-semantic-content.html

One could imagine putting one or more SCPs inside a Bag, combined with
using a mix of ORE and other vocabularies, for example. I felt it was
important to think about the problem in a way that wouldn't bind the
solution directly to BagIt or ORE, however, because it could then have
broader applicability.

Whether you like the technical approach or not, if you've got any
reactions I'd love to hear them either publicly or via email. I should
be clear, though, that this is very much in the experimental phase.
Just an idea I wanted to use to drum up some much-needed discussion in
this area.

- Chris

[Ed Summers]

Hi Chris,

Thanks for posting about Semantic Content Packages [1]. I've been a bit slow in getting around to reading it. Here are a few thoughts I had after reading. I hope you take these as an expression of my interest in what you are doing rather than criticisms :-)

- I like how the package metadata is bundled in .scpi so that you do not have to move data around on the file system in order to package it up. This is something some of us at the Library of Congress have talked informally about wanting to change in BagIt. It is also a behavior that developers are already familiar with in version control systems (Git, Mercurial, Subversion, etc). Incidentally, what's the "i" in .scpi for? 

- I like how every package must have a .scpi/id which contains a URI for the package. The alternative would be to have to monkey around around in .scp/graph.ttl to figure it out, but that could get messy. Is the statement about ASCII encoding really necessary, can't you just defer to RFC 3986 [2]?

- I like the use of Turtle RDF to describe the packages, since it really opens up the variety of assertions that can be made about the package. I think it also makes it possible to make explicit assertions about the contents of the package itself. This is something BagIt doesn't provide for, other than the ability to make an assertion about the fixity for a file in the package.  It also seems to be in line with some of the work being done at CDL [2] in their use of ORE. I wonder if CDL might have some comments about how well the approach has been working so far. As far as RDF goes Turtle is pretty readable--not as readable as ANVL [3], but I guess there has to be some cost to the flexibility.

- It looks like an empty graph.ttl file would be a valid Semantic Content Package? Are there a base set of assertions that need to be made to consider it "valid"?  This is one of the real strengths of BagIt IMHO. You don't have a valid bag unless you know" what the contents of the package are, and their checksums. For digital preservation purposes this is really important. Having a baseline understanding of what I can do with an SCP seems really important.

- Why not encourage people to subclass scp:Package and use rdf:type if they need to specialize what type of package they have instead of creating a new type property scp:type? Having two different type predicates for the resource seems awkward. If subclassing (and basic inferencing) isn't desired you could always have multiple rdf:type assertions.

- Did you create PackagePath intentionally instead of leaning on file:// protocol URIs?

- If one were to assert a checksum would it be made on the ByteStream or on the ContentLocation? Do you want to define a predicate for that in SCP or is there one people could use?

- Does it seem useful to be able to create packages whose contents are entirely ResolvableURIs? It doesn't look like scp:inPackage allows this.

- Does it seem reasonable to want a scp:hasBytestream predicate, similar to what ORE has in ore:aggregates and ore:isAggregatedBy?

- If the URI in .scpi/id is a HTTP URI, and it is resolved do you have any thoughts about what would come back? Would it be application/scp? Could it be the contents of .scp/graph.ttl with resolvable Bytestream URIs?

Like I said, I hope my questions above reflect that I think this is valuable work you are doing. It might be instructive to work on a few tools to see how easy it is to start using. Have you started using this scheme at all?  I think it might also be useful to sketch out some use cases for SCP. Why would you want SCP instead of BagIt or ORE for example?

//Ed

[1] http://www.ietf.org/rfc/rfc3986.txt
[2] https://wiki.ucop.edu/download/attachments/50790519/Merritt-object-modeling-latest.pdf

[3] https://wiki.ucop.edu/display/Curation/Anvl