Fedora's HTTP API: Memento for versioning
A. Soroka
There was general agreement that this move would be in alignment with our policy of adopting established or popular standards instead of “rolling our own”, that Memento is a thoroughly established standard, and that it would seem to support our use cases very well. There is a minor concern that there is currently no part of the Memento specification that covers the creation of versioned material, but Andrew Woods was able to speak with Herbert Van de Sompel (the leader of the Memento effort) about this at the CNI Fall Meeting, and he was very welcoming to the idea of expanding the specification in that direction.
We’d like to invite interested members of the community to comment on this idea. The tenor of the conversation was very positive: we saw no obvious problems with adopting Memento, and several advantages, but before any planning or work is done in this direction, we want to hear from you.
---
A. Soroka
The University of Virginia Library
[1] https://wiki.duraspace.org/display/FF/2016-01-07+-+Fedora+Tech+Meeting
[2] https://wiki.duraspace.org/display/FEDORA4x/RESTful+HTTP+API#RESTfulHTTPAPI-Versioning
[3] http://www.mementoweb.org/guide/quick-intro/
A. Soroka
The questions included:
1) To what extent do Fedora’s use cases require full (ACID [3]) transactions?
It seemed to some of us that of the traditional transaction guarantees, the use cases recorded for Fedora really demand only atomicity and durability. Reducing these guarantees in the (eventual) specifications of the Fedora APIs should reduce the burden on implementations considerably.
2) Would it be appropriate to mark transactions (in whatever form they are specified) as an “optional” feature in the Fedora API?
This would remove the burden entirely on implementations, but it raises the the question of how integral to common Fedora practices and workflows transactions or atomic operations are and will be. During the meeting, several people from the Islandora community remarked that current work on Islandora CLAW [4] is assuming the availability of multipart atomic operations from the Fedora API, and it’s not immediately clear how difficult it would be to remove that assumption. Marking this feature optional in the API specifications would mean that no framework that wanted to advertise compatibility with arbitrary Fedora implementations would rely on it being present in the repository API.
We invite interested parties in the community to comment on these questions. Does Fedora need full transactions, or would some simpler notion of “atomic batch operation” meet the use cases? In any case, is any such notion so indispensable that it must be included in all Fedora implementations, or should it be possible to do typical Fedora workflows in its absence? (This latter of course includes the possibility that middleware like Islandora CLAW or the nascent API-X project might host some such facility outside of the core repository APIs.)
---
A. Soroka
The University of Virginia Library
[1] https://wiki.duraspace.org/display/FF/2016-01-07+-+Fedora+Tech+Meeting
[3] https://en.wikipedia.org/wiki/ACID
[4] https://islandora-claw.github.io/CLAW/
Benjamin Armintor
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
Robert Sanderson
Tom Johnson
+1 from me too. (I'm one of the people Ben mentioned in his message).
I would be willing to contribute to any additions or supplements to the Memento spec, and could develop a second (Ruby RDF::LDP) implementation in parallel with any fedora development.
I think it would be good to advise the LDP community group if we have concrete plans to work on this. Ideally, this work could take place under the auspices of that group.
- Tom
Neil Jefferies
+1
Since I'm still on this!
Neil
Nick Ruest
+1 here.
-nruest
Neil Jefferies
an outcome of versioning.
It would be good to bundle a set of updates to different parts of an
object constituting a new version that either succeeds or fails.
Full ACID requirements across multiple objects I see as a symptom of
not properly changing mental gear from the SQL database table world to
the object/graph approach (queue indignant howls!).
However, I will concede at this point that I'm not sure we are really
ready for the pure object/graph play with the systems we currently have
in place. To do without transactions would mean delving much more into
rules and inference (e.g. to ensure that implicit reverse relationships
exist without requiring additional object updates).
So now I'm in two minds but "atomic batch operation" sounds like an
interesting approach.
Neil
Arif Shaon
Memento looks really promising, so +1 from UNSW as well.
On a separate but related note, since Memento is an output from the same community that is behind OAI-ORE and OAI-PMH, I wonder if there is work done on possible interoperability/integration between these protocols. At UNSW library, our Fedora3-based repositories use OAI-PMH services to enable harvesting of records by external repositories, and there are use cases that involve enabling access to different versions of a record via OAI-PMH and Fedora 3 API. I know this is a separate conversation, so I am just thinking out loud.
Cheers
Arif
Dr Arif Shaon
Lead Technical Officer, Library Repository Services, UNSW Library
Phone +61 2 9385 3088 | Fax +61 2 9385 8002 | a.s...@unsw.edu.au| http://www.library.unsw.edu.au
UNSW Library | UNSW Australia | UNSW Sydney NSW 2052 AUSTRALIA | CRICOS Provider Code: 00098G
From: fedora-c...@googlegroups.com [mailto:fedora-c...@googlegroups.com]
On Behalf Of Chad Mills
Sent: Friday, 8 January 2016 9:58 AM
To: fedora-c...@googlegroups.com
Cc: Fedora Tech
Subject: Re: [fedora-community] Re: [fedora-tech] Fedora's HTTP API: Memento for versioning
As an unbiased long time lurker, +1 as well.
--
Chad Mills
Rutgers University Libraries
Digital Library Architect
Ph: 848.932.5924
Cell: 732.309.8538
https://rucore.libraries.rutgers.edu/
On Jan 7, 2016 4:30 PM, Robert Sanderson <azar...@gmail.com> wrote:
A very biased (as Memento co-author) +1 :)
Rob
On Thu, Jan 7, 2016 at 12:03 PM, Benjamin Armintor <armi...@gmail.com> wrote:
Some colleagues and I started sketching out a proposal in this vein yesterday, in a happy coincidence. There are some questions, but in the main I think this is an approach to be enthusiastic about.
- Ben
On Thu, Jan 7, 2016 at 2:33 PM, A. Soroka <aj...@virginia.edu> wrote:
On today’s tech call [1] we discussed the possibility of moving Fedora’s HTTP API for versioning [2] to be an implementation of Memento [3], the well-known and widely-supported specification for publishing versioned content on the Web.
There was general agreement that this move would be in alignment with our policy of adopting established or popular standards instead of “rolling our own”, that Memento is a thoroughly established standard, and that it would seem to support our use cases very
well. There is a minor concern that there is currently no part of the Memento specification that covers the creation of versioned material, but Andrew Woods was able to speak with Herbert Van de Sompel (the leader of the Memento effort) about this at the CNI
Fall Meeting, and he was very welcoming to the idea of expanding the specification in that direction.
We’d like to invite interested members of the community to comment on this idea. The tenor of the conversation was very positive: we saw no obvious problems with adopting Memento, and several advantages, but before any planning or work is done in this direction,
we want to hear from you.
---
A. Soroka
The University of Virginia Library
[1]
https://wiki.duraspace.org/display/FF/2016-01-07+-+Fedora+Tech+Meeting
[2]
https://wiki.duraspace.org/display/FEDORA4x/RESTful+HTTP+API#RESTfulHTTPAPI-Versioning
[3] http://www.mementoweb.org/guide/quick-intro/
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
fedora-tech...@googlegroups.com.
To post to this group, send email to
fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
fedora-tech...@googlegroups.com.
To post to this group, send email to
fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
--
Rob Sanderson
Information Standards Advocate
Digital Library Systems and Services
Stanford, CA 94305
--
You received this message because you are subscribed to the Google Groups "Fedora Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
fedora-communi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Fedora Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
fedora-communi...@googlegroups.com.
har...@umich.edu
A. Soroka
Are you using any qualities of transactions other than atomicity? Would (in answer to my first question) atomic batch operations serve you as well?
No one has suggested removing transaction (or atomic operation) support from the current implementation. The question on hand is whether that support should be enshrined in future specifications (and if so, in what form) so that all future implementations must support it.
---
A. Soroka
The University of Virginia Library
Harshakumar Ummerpillai
You received this message because you are subscribed to a topic in the Google Groups "Fedora Tech" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fedora-tech/LAsEn46N5N0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
Tom Johnson
Taking atomicity and durability as assumed, can we clarify what consistency and/or isolation guarantees Fedora currently makes?
I don't see details on this in the transactions documentation link provided, and it's not clear to me what implementation burdens are at issue.
- Tom
Benjamin Armintor
In the LDP context, I'm not even sure what consistency means- I guess containment and membership triples?
A. Soroka
I’m not sure that LDP is relevant in this context.
---
A. Soroka
The University of Virginia Library
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-communi...@googlegroups.com.
Aaron Coburn
However, I would like to bring up some reasons why full ACID guarantees become problematic as part of the Fedora API specification. I am making a clear distinction between the Fedora specification (API) and the Fedora implementation (Modeshape).
As mentioned above, in a single-node context, transactions are not terribly difficult to implement with locks and mutexes, and with Modeshape, these transactions basically come "for free" with the JCR notion of Sessions. I would note that the Modeshape community has found that providing ACID guarantees in a distributed context, however, is proving very difficult, and this is largely why Modeshape 5 will effectively dispense with horizontal scalability, focusing instead on vertical (single-node) scalability. I would also reiterate that no one is suggesting removing full ACID guarantees from the Modeshape implementation of Fedora.
In contrast, for anyone who is interested in a horizontally scalable Fedora system (for example: a Fedora implementation that runs on top of Hadoop and HBase), developing such an implementation that requires full ACID guarantees becomes very non-trivial. HBase and other such distributed systems have a weak notion of consistency, and true independence is very difficult, if not impossible, to achieve (in fact, it undermines much of the performance gains that scaling out would provide).
By requiring that the Fedora API specification provides full ACID guarantees, it would mean that any future Fedora implementation (i.e. an implementation that doesn't use Modeshape) would need to implement those guarantees. If, however, the API specification has a weak notion of transactions, (that is, as Adam put it: "atomic batch operations"), then a horizontally distributed implementation of Fedora becomes significantly more feasible.
Regards,
Aaron
Benjamin Armintor
I assume the API spec takes recourse to LDP, and am thinking about what consistency means in that context. I am otherwise unsure about how ACID fits into the API spec, though the A & D seem relatively clear.
A. Soroka
Perhaps you are thinking of the documentation for the current implementation?
---
A. Soroka
The University of Virginia Library
Benjamin Armintor
Tom Johnson
The current implementation offers snapshot isolation (I’m not quite sure what Harsha means by "create a snapshot within a transaction”— snapshot isolation is _inherent_ to transactions) and a set of consistency guards inherited from ModeShape/JCR which the tech team discourages people from using, to avoid further dependency on ModeShape/JCR.
I’m not sure that LDP is relevant in this context.
In contrast, for anyone who is interested in a horizontally scalable Fedora system (for example: a Fedora implementation that runs on top of Hadoop and HBase), developing such an implementation that requires full ACID guarantees becomes very non-trivial. HBase and other such distributed systems have a weak notion of consistency, and true independence is very difficult, if not impossible, to achieve (in fact, it undermines much of the performance gains that scaling out would provide).
A. Soroka
Internal links in a Fedora repository could be system- or user-manged. In the first case, it should never be possible to use a sequence of API operations to move the repository into a state wherein system-managed internal links have lost integrity. In the second case, I don’t understand the current implementation to make any guarantees about the integrity of those links, but it at least formerly did, and that facility has been repeatedly requested (e.g. [1]), and users have gone as far as to penetrate the Fedora codebase and rely directly on JCR functions to that end.
I’m sure you’ll agree that the (to-be-written) Fedora API specifications should not be a thoughtless translation of the current faults and strengths of the current implementation. The question at hand is whether to consider consistency for the API spec, not for the current implementation. (If the API adopts some such notion, I suppose the current implementation would move to support it.)
---
A. Soroka
The University of Virginia Library
A. Soroka
That all having been said, I hope it’s very clear by now that I am not in favor of including any such consideration in any Fedora specification. I would prefer to restrict the functionality associated with Fedora transactions to atomicity and durability. On the other hand, it is perfectly clear that there are members of the community who would like to be able to establish notions of validity and consistency in their repositories.
---
A. Soroka
The University of Virginia Library
A. Soroka
> Since "ACID" has multiple interpretations and, in some cases, levels of support for each of its parts (besides often being discussed as though RDBMSs are the only relevant databases), it doesn't seem like a useful stand-in for a rigorous accounting of those guarantees.
---
A. Soroka
The University of Virginia Library
Tom Johnson
On Jan 8, 2016 10:17 AM, "A. Soroka" <aj...@virginia.edu> wrote:
>
> I’m not sure what interactions you are imagining between internal links and consistency for user-managed transactions.
>
- T1 begins, with snapshot isolation.
- Deletion of R1 is added to T1.
- T2 begins, with snapshot isolation.
- An update to R2 is added to T2, containing a triple of the form <R2> <P> <R1>.
- T1 commits.
- T2 commits.
If we guarantee internal link integrity, T2 should fail as inconsistent.
> Internal links in a Fedora repository could be system- or user-manged. In the first case, it should never be possible to use a sequence of API operations to move the repository into a state wherein system-managed internal links have lost integrity. In the second case, I don’t understand the current implementation to make any guarantees about the integrity of those links, but it at least formerly did, and that facility has been repeatedly requested (e.g. [1]), and users have gone as far as to penetrate the Fedora codebase and rely directly on JCR functions to that end.
>
I don't follow much of this, but if internal link consistency is no longer a requirement of the API, obviously it won't effect transactions.
> I’m sure you’ll agree that the (to-be-written) Fedora API specifications should not be a thoughtless translation of the current faults and strengths of the current implementation. The question at hand is whether to consider consistency for the API spec, not for the current implementation. (If the API adopts some such notion, I suppose the current implementation would move to support it.)
>
> ---
> A. Soroka
> The University of Virginia Library
>
> [1] https://wiki.duraspace.org/display/FF/Optional+validation
>
> > On Jan 8, 2016, at 10:00 AM, Tom Johnson <johns...@gmail.com> wrote:
> >
> > A. Soroka said:
> >
> > The current implementation offers snapshot isolation (I’m not quite sure what Harsha means by "create a snapshot within a transaction”— snapshot isolation is _inherent_ to transactions) and a set of consistency guards inherited from ModeShape/JCR which the tech team discourages people from using, to avoid further dependency on ModeShape/JCR.
> >
> > I’m not sure that LDP is relevant in this context.
> >
> > Fedora makes certain guarantees about the integrity of internal links; assuming those guarantees are retained in the spec, it seems like consistency can't be dispensed with, right? Or are those the consistency guards that you mention?
>
A. Soroka
---
A. Soroka
The University of Virginia Library
A. Soroka
The facts are these: the current implementation, at various points in its history, sometime has and sometimes has not supported this kind of integrity. Many users have expressed interest in it or actually did use it. We now need to decide whether or not to require it as a feature of the Fedora API.
> I don't follow much of this, but if internal link consistency is no longer a requirement of the API, obviously it won't effect transactions.
—
The University of Virginia Library
Tom Johnson
> If we guarantee internal link integrity, T2 should fail as inconsistent.
Indeed, if we decide to adopt some notion of consistency, and if that notion includes this kind of internal link integrity, that transaction should fail. I’m not sure what you are arguing here. Are you suggesting that some kind of consistency _should_ or _should not_ be part of the Fedora API, and if the former, what kind of consistency are you suggesting that should be?
The facts are these: the current implementation, at various points in its history, sometime has and sometimes has not supported this kind of integrity. Many users have expressed interest in it or actually did use it. We now need to decide whether or not to require it as a feature of the Fedora API.
> I don't follow much of this, but if internal link consistency is no longer a requirement of the API, obviously it won't effect transactions.
I think you may be confused about the issue under discussion. The question on the table is exactly whether some notion of consistency _is_ going to be a requirement of the API for transactions.
—
Arif Shaon
Happy to provide information if necessary.
Regards
Dr Arif Shaon
Lead Technical Officer, Library Repository Services, UNSW Library
Phone +61 2 9385 3088 | Fax +61 2 9385 8002 | a.s...@unsw.edu.au| http://www.library.unsw.edu.au
UNSW Library | UNSW Australia | UNSW Sydney NSW 2052 AUSTRALIA | CRICOS Provider Code: 00098G
A. Soroka
https://wiki.duraspace.org/display/FF/University+of+Virginia+-+Applications+can+be+easily+built+to+work+against+fedora
https://wiki.duraspace.org/display/FF/Transaction+Use+Case
so the more the merrier!
---
A. Soroka
The University of Virginia Library
A. Soroka
In the absence of any objection between now and then, we will move forward to create a plan for shifting Fedora’s HTTP API for versioning to a presentation of Memento in the LDP context. We will be examining related work done in the Apache Marmotta project, we will be in contact with the Memento community (particularly on the question of HTTP action for creating new versions) and of course, anyone in the community with an interest is invited to participate.
[1] https://wiki.duraspace.org/display/FF/2016-01-14+-+Fedora+Tech+Meeting#id-2016-01-14-FedoraTechMeeting-memento
---
A. Soroka
The University of Virginia Library
> On today’s tech call [1] we discussed the possibility of moving Fedora’s HTTP API for versioning [2] to be an implementation of Memento [3], the well-known and widely-supported specification for publishing versioned content on the Web.
>
> There was general agreement that this move would be in alignment with our policy of adopting established or popular standards instead of “rolling our own”, that Memento is a thoroughly established standard, and that it would seem to support our use cases very well. There is a minor concern that there is currently no part of the Memento specification that covers the creation of versioned material, but Andrew Woods was able to speak with Herbert Van de Sompel (the leader of the Memento effort) about this at the CNI Fall Meeting, and he was very welcoming to the idea of expanding the specification in that direction.
>
> We’d like to invite interested members of the community to comment on this idea. The tenor of the conversation was very positive: we saw no obvious problems with adopting Memento, and several advantages, but before any planning or work is done in this direction, we want to hear from you.
>
> A. Soroka
> The University of Virginia Library
>
> [2] https://wiki.duraspace.org/display/FEDORA4x/RESTful+HTTP+API#RESTfulHTTPAPI-Versioning
> [3] http://www.mementoweb.org/guide/quick-intro/
>
David Wilcox
Chad Mills
As an unbiased long time lurker, +1 as well.
--
Chad Mills
Rutgers University Libraries
Digital Library Architect
Ph: 848.932.5924
Cell: 732.309.8538
https://rucore.libraries.rutgers.edu/
A very biased (as Memento co-author) +1 :)Rob
On Thu, Jan 7, 2016 at 12:03 PM, Benjamin Armintor <armi...@gmail.com> wrote:
Some colleagues and I started sketching out a proposal in this vein yesterday, in a happy coincidence. There are some questions, but in the main I think this is an approach to be enthusiastic about.- Ben
On Thu, Jan 7, 2016 at 2:33 PM, A. Soroka <aj...@virginia.edu> wrote:
On today’s tech call [1] we discussed the possibility of moving Fedora’s HTTP API for versioning [2] to be an implementation of Memento [3], the well-known and widely-supported specification for publishing versioned content on the Web.
There was general agreement that this move would be in alignment with our policy of adopting established or popular standards instead of “rolling our own”, that Memento is a thoroughly established standard, and that it would seem to support our use cases very well. There is a minor concern that there is currently no part of the Memento specification that covers the creation of versioned material, but Andrew Woods was able to speak with Herbert Van de Sompel (the leader of the Memento effort) about this at the CNI Fall Meeting, and he was very welcoming to the idea of expanding the specification in that direction.
We’d like to invite interested members of the community to comment on this idea. The tenor of the conversation was very positive: we saw no obvious problems with adopting Memento, and several advantages, but before any planning or work is done in this direction, we want to hear from you.
---
A. Soroka
The University of Virginia Library
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.
--
Rob SandersonInformation Standards AdvocateDigital Library Systems and ServicesStanford, CA 94305
--
Aaron Birkland
Neil Jefferies
We shouldn't need to - if we access the object via memento we have an implicit date time that we can use traverse the linked resources also using memento - without having to update the links in situ (good for preservation reasons)
If the links are external and don't support memento then we just have to link to the base URI.
Neil
On 2016-01-21 14:30, Aaron Birkland wrote:
I'm very much +1 on this idea as well. There may be some interesting questions related to the contents of versioned resources - for example, would versions essentially retain their current semantics as a sort of 'snapshot' of an entire tree of resources, with every link to a repository resource replaced with a link to its memento within a version?-Aaron
--