Fedora API specifications

[Simeon Warner]

Hi all,

Apologies for missing the last call where the purpose of the spec was
discussed, it seems that was useful discussion. Going through the notes
from the call [1], the previous discussion [2] and the old wiki pages
with rationale [3] I think we are converging on two broad goals:

1. better align Fedora with existing standards, and reuse where possible
(... to better focus effort of community, better support tool reuse, etc.)

2. provide an opportunity for alternate implementations (... to provide
for lightweight test fixtures, or alternate implementations suited for
specific service profiles or environments)

I think the effort is doing very well on 1, but much less well on 2. For
2 to be meaningful we really are talking about swappable
implementations, with the generalization that there might be a checklist
of features/behaviors required in order to be swappable for a given
application.

The first implication of swappable is that there would need to be at
least one more specification (2 if fixity becomes a spec):

a) Fedora API -- https://fcrepo.github.io/fcrepo-specification/

b) Atomic operations --
https://github.com/fcrepo/fcrepo-specification-atomic-operations

c) maybe... Fixity spec --
https://github.com/fcrepo/fcrepo-specification/issues/90

d) "Other Fedora API Behaviors Spec" -- covering things like storage
pattern, slug handling, URI patterns, single subject restrictions (or
not), and more

Thus, if Derby were to evolve into a useful test fixture for Fedora5, it
would need implement all of these specifications (a,b,c,d), and
hopefully the specs would be complete enough for it meet that need
without reference to "yet more Fedora things not in the spec".

If the Hydra framework does not use atomic batch operations or
persistence fixity (say), then the checklist of specs needed to be
support a Hydra app would be (a) and (d).

To me this line of thinking implies two things:

i) There will be a need to write (d), the "Other Fedora API Behaviors
Spec" (or specs, probably with a better name)

ii) It is desirable for (d) to be as small as possible, but not at the
expense of sacrificing specificity. It really must include everything
about "what the Fedora API does" (e.g. including the current "RESTful
HTTP API" documentation [4]) that is not in the other specs. For example
that the root container is at /rest/, the single subject rule, how/when
slug requests are honored [5], etc.. However, anything behavior that
might be usefully built into another implementation (such as Trellis or
Cavendish) would be best split out into a spec along the lines of Atomic
Batch Operations.

Under this scenario we have a plausible way to implement either
lightweight test fixtures that are drop-in replacements, or to
understand through a checklist of specs/sections whether a particular
alternate implementations will support a given application. The finesse
with which specs/sections are split up will determine how clean that
checklist is. Ideally there would also a test suite for each
spec/section to verify support and interoperability across implementations.

Attached is a picture I drew to help think about this.

2c,
Simeon


[1] https://wiki.duraspace.org/display/FF/2017-04-13+-+Fedora+Tech+Meeting

[2] https://wiki.duraspace.org/display/FF/2017-03-23+-+Fedora+Tech+Meeting

[3] https://wiki.duraspace.org/display/FEDORAAPI/Fedora+Specification

[4] https://wiki.duraspace.org/display/FEDORA4x/RESTful+HTTP+API

[5] https://github.com/fcrepo4-labs/derby/issues/6

[west...@umd.edu]

Simeon,

Thank you for putting this together. It is very helpful in clarifying where things stand and pointing out possible ways to reconcile the different viewpoints.  If my reading of the tenor of the Github issue discussion on Fixity is correct [1], then fixity appears likely to be removed from the core, in which case we at Maryland would be strongly in favor of the development of your option 'c', where fixity becomes its own spec along the lines of batch atomic operations.

Josh

[1] https://github.com/fcrepo/fcrepo-specification/issues/90

[Aaron Coburn]

> Apologies for missing the last call where the purpose of the spec was discussed, it seems that was useful discussion. Going through the notes from the call [1], the previous discussion [2] and the old wiki pages with rationale [3] I think we are converging on two broad goals:
>
> 1. better align Fedora with existing standards, and reuse where possible (... to better focus effort of community, better support tool reuse, etc.)

Yes, I completely agree.

> 2. provide an opportunity for alternate implementations (... to provide for lightweight test fixtures, or alternate implementations suited for specific service profiles or environments)

Again, I agree.

> I think the effort is doing very well on 1, but much less well on 2. For 2 to be meaningful we really are talking about swappable implementations, with the generalization that there might be a checklist of features/behaviors required in order to be swappable for a given application.

On this point, I very much disagree. And I will push back vigorously on the point of "swappable implementations" being a goal of the specification effort.

The underlying question here, to my mind, is this: is the "Fedora specification" a specification or the description of an application?

Another way of saying this is: is the purpose of the specification to allow for the creation of Fedora4 clones or is it to allow for innovation?

If there is widespread belief that Fedora4 is the end-all-and-be-all of repository applications, then having the "spec" be a means by which alternate clones can be written is a fine goal. I do not hold this opinion. I see many ways that Fedora4 is lacking both in implementation and in design. If the "spec" is truly a specification, then it will be an open document that allows for a wide range of interpretations of various behaviors. And as such, implementations will most certainly not be "swappable".

In truth, "swappable" is a very slippery concept. And it is meaningless to talk about "swappable servers" without talking about the corresponding clients. For a server can only be called "swappable" in the context of a particular client or set of clients. However, as long as Fedora's interface is LDP and HTTP, then the "clients" for Fedora constitute an unbounded set of HTTP clients. I will assert that making an implementation "swappable" for an unbounded set of potential clients is, for all intents and purposes, an impossible goal.

It has already been pointed out that LDP imposes very few restrictions on the behaviors of servers. Even though the Fedora specification tightens some of the language here, there is still a broad range of permissible -- though (depending on the client) incompatible -- behaviors that can be part of an implementation. Here are just a few ways in which conforming Fedora implementations could be incompatible:

1. Implementation A users server-managed triples while Implementation B puts those into headers
2. Implementation A has a single-subject restriction while Implementation B doesn't
3. Implementation A supports the creation of non-Container LDP-RSs while implementation B doesn't
4. Implementation A supports blank nodes while Implementation B skolemizes them
5. Implementation A requires Link: rel="type" headers for the creation of new resources while implementation B uses a heuristic based on MIMEtype
6. Implementation A supports client-controlled versioning while implementation B does only server-managed versioning
7. Implementation A enforces referential integrity while implementation B doesn't
8. Implementation A allows uncontained PUT while implementation B doesn't
9. Implementation A supports rdf/xml while implementation B doesn't
10. Implementation A deletes resources recursively, while implementation B has a different model for deletes

I can go on and on with such examples. As we know from the Fedora4 issue related to the removal of the "xsd:string" datatype on string literals, there is disagreement about what is a "breaking/incompatible change" and what isn't. This will only get worse across multiple implementations if "swappability" is a goal.

Another example is the support of "message/external-body". This is a big weakness of the specification at present. In fact, IMO, this is fundamentally incompatible with LDP. As defined by RFC 2017, this is a mechanism for email messages to switch protocols for the message body; that is, message headers may use a pop3: or imap: protocol, but the body can be fetched from a different protocol, e.g.: gopher:, ftp:, file:, or http:. The idea of "protocol-switching" does not in any way fit with LDP as a HTTP/1.1 server.

From what is described below, it seems as though the direction of the specification writers is to specify more (not less) of the Fedora behaviors. I think this is a bad approach, and doing so will the death knell not just for the Fedora specification but for the Fedora project as a whole.

If anything, we need more innovation in the area of repository implementations, not less. And over-specifying "what is Fedora" will result in less innovation.

If it were up to me (which is clearly isn't), the entire specification would consist of about 5 sentences:

1. A Fedora server MUST be a conforming LDP server.
2. A Fedora server MUST provide time-based access to resources according to RFC 7089 (Memento)
3. A Fedora server SHOULD support instance digests on resources according to RFC 3230
4. A Fedora server that enforces authorization MUST use WebAC as defined by the Solid specification
5. For any resource that is changed, a Fedora server MUST make a notification available of that change; notifications MUST conform to the ActivityStreams specification.

There is clearly considerable disagreement on the goal of the specification effort. In my opinion, this is a good thing, and the specification itself should allow for that type of disagreement. Again, in my opinion, the specification can accommodate these multiple perspectives only by not over-specifying behavior.

Regards,
Aaron

> The first implication of swappable is that there would need to be at least one more specification (2 if fixity becomes a spec):
>
> a) Fedora API -- https://fcrepo.github.io/fcrepo-specification/
>
> b) Atomic operations -- https://github.com/fcrepo/fcrepo-specification-atomic-operations
>
> c) maybe... Fixity spec -- https://github.com/fcrepo/fcrepo-specification/issues/90
>
> d) "Other Fedora API Behaviors Spec" -- covering things like storage pattern, slug handling, URI patterns, single subject restrictions (or not), and more
>
> Thus, if Derby were to evolve into a useful test fixture for Fedora5, it would need implement all of these specifications (a,b,c,d), and hopefully the specs would be complete enough for it meet that need without reference to "yet more Fedora things not in the spec".
>
> If the Hydra framework does not use atomic batch operations or persistence fixity (say), then the checklist of specs needed to be support a Hydra app would be (a) and (d).
>
> To me this line of thinking implies two things:
>
> i) There will be a need to write (d), the "Other Fedora API Behaviors Spec" (or specs, probably with a better name)
>
> ii) It is desirable for (d) to be as small as possible, but not at the expense of sacrificing specificity. It really must include everything about "what the Fedora API does" (e.g. including the current "RESTful HTTP API" documentation [4]) that is not in the other specs. For example that the root container is at /rest/, the single subject rule, how/when slug requests are honored [5], etc.. However, anything behavior that might be usefully built into another implementation (such as Trellis or Cavendish) would be best split out into a spec along the lines of Atomic Batch Operations.
>
> Under this scenario we have a plausible way to implement either lightweight test fixtures that are drop-in replacements, or to understand through a checklist of specs/sections whether a particular alternate implementations will support a given application. The finesse with which specs/sections are split up will determine how clean that checklist is. Ideally there would also a test suite for each spec/section to verify support and interoperability across implementations.
>
> Attached is a picture I drew to help think about this.
>
> 2c,
> Simeon
>
>
> [1] https://wiki.duraspace.org/display/FF/2017-04-13+-+Fedora+Tech+Meeting
>
> [2] https://wiki.duraspace.org/display/FF/2017-03-23+-+Fedora+Tech+Meeting
>
> [3] https://wiki.duraspace.org/display/FEDORAAPI/Fedora+Specification
>
> [4] https://wiki.duraspace.org/display/FEDORA4x/RESTful+HTTP+API
>
> [5] https://github.com/fcrepo4-labs/derby/issues/6
>

> --
> You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.
> <2017-04_fedora_dependencies.pdf>

[soro...@gmail.com]

I agree with what Aaron says here. A few years ago, I would have disagreed strongly.

In the meanwhile, I have realized that I began working on the API spec with a badly mistaken assumption: that there is still a single model at the heart of Fedora activity and apparent in it. (A model descended from the CMA of Fedora 3 and thence from the original ideas with which the project started.) That's not the case.

Let me wax a bit pretentious (even for me) and talk about The Future. I can imagine a number of different futures for Fedora. In some, there is a common data model again and it becomes useful to work towards common practice. In some, there are several data models, each popular within different segments of a larger community (e.g. Hydra-using sites might select for PCDM). In some, there is a powerful diversity of models (clowder of models?), perhaps driven by domain-specific interests (e.g. bioinformatics) or even diversity of modeling within any given repository. (In fact, part of the attraction of RDF technology is precisely that last possibility.)

In any event, I believe that the most important ideas that Fedora ever had are these: durability is the preservation of meaning (as opposed to fixity, which is the preservation of form). It cannot occur without shared, transparent, (ideally machine-actionable) semantics. And it emerges from fine-grained modeling of content (using relationships between small objects to create larger ones).

From that point of view, it's not meaningful to talk about Fedora repositories that are "interoperable" if they don't share common data modeling, and since I don't believe that we start from that point, it's not useful to talk about interoperability as a goal, _unless_ we are willing to discount all but the first of those possible futures I described. I'm not willing to do that. Not at all. In fact, I suspect quite strongly that the best future and the one for which we are headed is the last one. But as far as we know now, choosing for any of those futures is a good way to guess wrong. The right choice, then, is to constrain the spec no more closely than will permit investigating further, and that's why a "loose" style is appropriate for it now.

---
A. Soroka
The University of Virginia Library



> Begin forwarded message:

---
A. Soroka
The University of Virginia Library

[A. Soroka]

I agree with what Aaron says here. A few years ago, I would have disagreed strongly.

In the meanwhile, I have realized that I began working on the API spec with a badly mistaken assumption: that there was still a single model at the heart of Fedora activity and apparent in it. (A model descended from the CMA of Fedora 3 and thence from the original ideas with which the project started.) That's not the case.

Let me wax a bit pretentious (even for me) and talk about The Future. I can imagine a number of different futures for Fedora. In some, there is a common data model again and it becomes useful to work towards common practice. In some, there are several data models, each popular within different segments of a larger community (e.g. Hydra-using sites might select for PCDM). In some, there is a powerful diversity of models (clowder of models?), perhaps driven by specific interests (e.g. bioinformatics) or even diversity of modeling within any given repository. (In fact, part of the attraction of RDF technology is precisely that last possibility.)

In any event, I believe that the most important ideas that Fedora ever had are these: durability is the preservation of meaning (as opposed to fixity, which is the preservation of form). It cannot occur without shared, transparent, (ideally machine-actionable) semantics. And it emerges from fine-grained modeling of content (using relationships between small objects to create larger ones).

From that point of view, it's not meaningful to talk about Fedora repositories that are "interoperable" if they don't share common data modeling, and since I don't believe that we start from that point, it's not useful to talk about interoperability as a goal, _unless_ we are willing to discount all but the first of those possible futures I described. I'm not willing to do that. Not at all. In fact, I suspect quite strongly that the best future and the one for which we are headed is the last one. But as far as we know now, choosing for any of those futures is a good way to guess wrong. The right choice, then, is to constrain the spec no more closely than will permit investigating further, and that's why a "loose" style is appropriate for it for now.

---
A. Soroka
The University of Virginia Library

[Trey Pendragon]

So, I want to start this off by saying I've got a lot less experience here with specifications than the people replying here, so feel free to take this as ignorance of the goal.

One of the big value propositions of all the time sunk into this API specification was that by the end I'd be able to have one client in a given language but multiple backends with different opinions on things like performance, distributed storage, indexing, etc. Effectively we as a Hydra application would be able to push those decisions to a middleware layer with a common interaction criteria, possibly shared with other projects or institutions.

However, what I'm hearing now is that's NOT the goal. Rather, the specification should be flexible enough for experimentation while still adhering to the "letter of the law". Basically, incompatibility between clients as a feature. That seems problematic to me, but we can work with that if we must. I do want to make a couple points though:

1) There's no reason that the specification you write now is gospel. I think we've proven we're (as implementors) more than happy to do whatever we need to in order to support a new major version of Fedora, should opinions on interaction models change. All that's important to us is that if I have a Fedora 5 server, my Fedora 5 client works, and that actually means something.

2) "However, as long as Fedora's interface is LDP and HTTP, then the "clients" for Fedora constitute an unbounded set of HTTP clients. I will assert that making an implementation "swappable" for an unbounded set of potential clients is, for all intents and purposes, an impossible goal." I really don't understand this. Sure, HTTP has a bunch of clients, but in general they all work. There are multiple implementations of the Amazon S3 specification, which works via HTTP, and somehow all the clients work with it just fine. We're not Amazon, but I think "impossible goal" is very strong wording for this.

If the specification ends up heading towards obscurity over precision, then maybe there's a good reason for that. However, it does mean arguments I've heard for NOT writing complicated in-app architectures for swappable backends and rather writing that as Fedora implementations hold little if any water, as "Fedora" ends up having little to no meaning for incompatibility.

Trey

[Benjamin J. Armintor]

There's a lot to process in this thread, so forgive the brevity of this reply - I'll come back to it as soon as I can. For now:

1. Thanks for this Simeon- this is a very useful summary.

2. I think I agree with Aaron, with the exception of interpreting RFC 2017 as a protocol switch (I read it as an external reference), but I'm not sure that the direction of the effort to date hasn't been towards expressing those 5 sentences as verifiable behaviors as best we can (and non-normative text about why we can't when we can't). I also think I have a different understanding of swappability, but I'm not sure it matters if we agree about the earlier points.

3. I think I agree with Adam about this understanding of repositories as systems or applications, but I also think there's still *a* (not *the*) useful thing to be specified that supports the various models Adam describes (cf Aaron's 5 sentences).

4. Like Trey, I also have been making the argument to swathes of the GLAM technology community that we can agree on some practices-as-network-interactions around which we build solutions. Pointing only to higher levels means that we as the Fedora community share goals; that's good! But there are also communities of practice, and within those communities of implementation, and they have needs for what Trey describes. That is to say, if the Fedora community is uninterested in what Trey discusses as an achievable goal then this kind of specification needs to be remanded to the communities of implementation, and there's a reckoning to be made in that case for agreeing that (hypothetically) Fedora is not the thing Islandora and Hydra have in common, or indeed what different Hydra applications have in common.

To date, Fedora (or something that acts like it) has been the boundary that defines shareable tooling for Columbia, Princeton, BPL, etc. If we remain committed to sharing tools, something needs to define that boundary. We've talked about having further refinements of the spec in the application communities, so maybe this is just a question of how much we need to ask e.g. the Hydra Architecture WG to codify more expectations in a "suitability test kit" or similar, but: I still think there's a useful level of commonality in the documents we're writing, and I'd like this effort to be held in common someplace like this group.

- Ben

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Andrew Woods]

Thank you, Simeon, for articulating a model that represents units of work and intent around the Fedora API Specification effort. This is extremely constructive.

Over the past year, the differences in understanding of the purpose of the specification have pulled us away from the original rationale and objectives. I think it is important to take this opportunity to recenter the effort.

https://wiki.duraspace.org/display/FEDORAAPI/Fedora+Specification

As mentioned in this thread, Fedora represents a boundary of functional responsibility and therefore a focal point for community collaboration and a common foundation for building applications.

One of the primary goals of the specification is to draw a clear line of abstraction so that client applications can be built against a stable, versioned API while allowing what is behind that API to evolve. The guidance in the specification must be clear enough to enable compliance test suites.

Will Fedora implementations be fully swappable without requiring fine adjustment within clients, likely not. However, if the specification is so loose that clients are effectively forced to build against the specific APIs of individual implementations, then the broader Fedora community will likewise be fragmented along those same lines.

That said, this is much less of a technical discussion than a community-direction one.

I am bringing the Fedora Leadership into this thread and will be discussing the Fedora API Specification mandate next Tuesday on the Fedora Steering meeting.

Regards,

Andrew

[Robin Lindley]

Andrew - hear, hear and I look forward to the discussion. 

Robin 

You received this message because you are subscribed to the Google Groups "Fedora Leaders" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-leader...@googlegroups.com.

[Joshua Westgard]

I too very much look forward to this discussion and hope for a resolution.  It seems to me that we are in danger of deconstructing ourselves right out of what up until now has been a very important community of practice.  

I'm not sure how one maintains a community on the basis of a specification without specifics and various Fedoras without anything in common beyond implementing LDP and a few other standards.  I trust that's not where we're heading and look forward to finding common ground on which to build, which is what I hoped the API spec would provide.

Josh

[soro...@gmail.com]

I'm going to bring together a few threads in this discussion, to avoid multiplying emails any more than needed.

---
A. Soroka
The University of Virginia Library

> On Apr 19, 2017, at 10:02 PM, Andrew Woods <awo...@duraspace.org> wrote:

...

> Over the past year, the differences in understanding of the purpose of the specification have pulled us away from the original rationale and objectives. I think it is important to take this opportunity to recenter the effort.
> https://wiki.duraspace.org/display/FEDORAAPI/Fedora+Specification

I think it would be a good deal more accurate to say that in the several years in which work has gone on for the specification, discussion and investigation have changed our understanding of what is possible and what is practical to accomplish by writing down specifications. This is as it should be. It would be very disappointing if we worked hard for several years on this effort and arrived at the end with the same understanding with which we started.

At the beginning, I would have argued for a much more closely written specification than we now have. Thinking and discussion with everyone working on the spec (particularly Ben Armintor) have shown me what a mistake that would have been. Part of what I've learnt is that to the extent that Fedora invents its own API and specifies it, the Fedora community will have to pay the costs of maintaining special software to implement and interact with its own resources. To the extent that we avoid so doing, we can use off-the-shelf parts and participate in (_much_) larger communities.

Another way to put this: interoperability is not confined within the Fedora community. It's just as much as question of how Fedora integrates into larger software ecosystems and information architectures. The more closely specified Fedora becomes, the more we trade intramural for extramural "swappability".

> On Wed, Apr 19, 2017 at 11:56 AM, Trey Pendragon <tre...@gmail.com> wrote:

...

> Sure, HTTP has a bunch of clients, but in general they all work. There are multiple implementations of the Amazon S3 specification, which works via HTTP, and somehow all the clients work with it just fine. We're not Amazon, but I think "impossible goal" is very strong wording for this.

This is _such_ a good point. Without any official compliance test suite or imprimatur, the many, many HTTP clients in the world all do manage to work more-or-less properly. This is mostly because HTTP is not over-specified.

> On Apr 19, 2017, at 5:56 PM, Benjamin J. Armintor <ba2...@columbia.edu> wrote:
...

> 3. I think I agree with Adam about this understanding of repositories as systems or applications, but I also think there's still *a* (not *the*) useful thing to be specified that supports the various models Adam describes (cf Aaron's 5 sentences).

...

> To date, Fedora (or something that acts like it) has been the boundary that defines shareable tooling for Columbia, Princeton, BPL, etc. If we remain committed to sharing tools, something needs to define that boundary. We've talked about having further refinements of the spec in the application communities, so maybe this is just a question of how much we need to ask e.g. the Hydra Architecture WG to codify more expectations in a "suitability test kit" or similar, but: I still think there's a useful level of commonality in the documents we're writing, and I'd like this effort to be held in common someplace like this group.

Ben is pointing the way forward here. We have been discussing this issue as though every part of it were a "global" matter, in the sense of community-wide all-or-nothing. I am increasingly sure that this is a mistake. A concern in hand for many people clearly _is_ implementation "swappability", and the correct place to address that valid concern is not at the high-level information architecture (Fedora itself) but organically, out of communities of _implementation_ (e.g. Hydra, Islandora, etc.).

There is nothing preventing us from maintaining a crisp and quickly-understood and not-over-specified Fedora API along the lines suggested by Aaron C. (and with which I agree), while the (sub)communities that cohere around particular implementation platforms work out the refinements that lead to shareable tooling. If we want to formalize that kind of activity in a community-wide way, we can develop a notion of "profile" for the specification activity as a whole. That's a common and straightforward move. It seems much more sensible to me to begin to discuss tool sharing within communities that share _most_ of their tools, instead of in a larger community that shares only _one_ tool. As commonalities are discovered and invented, they can percolate up and out.

ajs6f

[Ben Cail]

On 04/20/2017 09:44 AM, soro...@gmail.com wrote:
> Ben is pointing the way forward here. We have been discussing this
> issue as though every part of it were a "global" matter, in the sense
> of community-wide all-or-nothing. I am increasingly sure that this is
> a mistake. A concern in hand for many people clearly _is_
> implementation "swappability", and the correct place to address that
> valid concern is not at the high-level information architecture
> (Fedora itself) but organically, out of communities of
> _implementation_ (e.g. Hydra, Islandora, etc.).
> There is nothing preventing us from maintaining a crisp and quickly-understood and not-over-specified Fedora API along the lines suggested by Aaron C. (and with which I agree), while the (sub)communities that cohere around particular implementation platforms work out the refinements that lead to shareable tooling. If we want to formalize that kind of activity in a community-wide way, we can develop a notion of "profile" for the specification activity as a whole. That's a common and straightforward move. It seems much more sensible to me to begin to discuss tool sharing within communities that share _most_ of their tools, instead of in a larger community that shares only _one_ tool. As commonalities are discovered and invented, they can percolate up and out.

Does this mean that the Islandora community would have its own Fedora
implementation, and Hydra would have its own Fedora implementation, and
others outside those two communities would pick/write some Fedora that
works for them? Then Hydra code would work with Hydra-Fedora, and
Islandora code would work with Islandora-Fedora, and each wouldn't be
guaranteed to work with any other Fedora, right?. What's the benefit in
calling these various implementations "Fedora"?

-different Ben

[Esmé Cowles]

Ben C.-

I hope that's not where we wind up. I would very much like Hydra to work with several different Fedora implementations (Modeshape, Cavendish, Derby, and hopefully others like Trellis and Static-LDP) and have adopters able to switch between them depending on their needs. And I would hope that Islandora would work with some or all of those same implementations.

There may be some clients that have very specific needs that only one Fedora implementation meets, but I hope that's a rare thing and not the norm.

-Esmé

[soro...@gmail.com]

> On Apr 20, 2017, at 10:50 AM, Esmé Cowles <esco...@ticklefish.org> wrote:
>
> Ben C.-
>
> I hope that's not where we wind up. I would very much like Hydra to work with several different Fedora implementations (Modeshape, Cavendish, Derby, and hopefully others like Trellis and Static-LDP) and have adopters able to switch between them depending on their needs. And I would hope that Islandora would work with some or all of those same implementations.
>
> There may be some clients that have very specific needs that only one Fedora implementation meets, but I hope that's a rare thing and not the norm.
>
> -Esmé

I'm not sure why that _would_ happen, unless it turns out that Hydra and Islandora and other sub-communities genuinely have radically different needs, which doesn't seem very likely to me.

What seems far more likely to me is that we would end up with a much bigger "tent" for Fedora implementations and a much bigger selection of implementations from which any given community could choose those that satisfy those needs that _are_ particular to it.

As has become abundantly clear from public remarks by early implementors like Aaron Coburn and Ben Armintor, not every attempt at implementation is going to accomplish or even _try_ to accomplish every piece of the spec _we have already written_. We've already chosen to / had to split off pieces of specification that were once considered core (e.g atomic operations, perhaps fixity as well). I don't think anyone is suggesting that either 1) perfect, thoughtless interchangeability is a realistic goal or that 2) every single site or subcommunity should have a unique implementation of its own.

What I am suggesting is that we must write and share different kinds of specification in the appropriate communities, and that means we cannot ignore the differences between the Fedora community (which has grown by sharing various information architectures) and communities of implementation within it. It's not unimportant for (say) Hydra and Islandora to be able to share implementations. It is more _immediately_ important for Hydra to be able to share implementations within itself (ditto other communities) and what's more, it's a good place to start in order to get to a world where Hydra and Islandora et al. _can_ share implementations. That kind of "bottom-up" growth will let us abstract over real experience to develop interoperability, as opposed to mandating it. That's what leads me to suggest keeping already-built consensus for the whole community in the Fedora spec and keeping assumptions about profiled behavior in some system of profiles (which could evolve much more easily and quickly than the "upper" document and which certainly should feed further revision thereof).

---
A. Soroka
The University of Virginia Library

[Aaron Coburn]

Hi, Graham,

>> 1. A Fedora server MUST be a conforming LDP server.

>> 5. For any resource that is changed, a Fedora server MUST make a notification available of that change; notifications MUST conform to the ActivityStreams specification.
>

> Forgive me here, but I’ve got a bunch of loosely scanned information floating around my head, and this also relates to something else I’ve been considering. So, a question:
>
> If you are specifying that something must be a conforming LDP server, why would you specify that notifications must conform to the ActivityStreams specification, rather than Linked Data Notifications?
>
> https://www.w3.org/TR/ldn/

Linked Data Notifications is great (I've written both a sender and consumer and have contributed -- in a minor way -- to that specification). LDN is primarily about defining an interaction model, defining where notifications are sent (i.e. to an ldp:inbox) and how that inbox is discovered. LDN is really not about the _content_ of the message (other than specifying that the content be RDF, specifically JSON-LD).

Activity Streams (which is entirely compatible with LDN) defines a specific payload format for these messages. Even AS/2.0 provides a fair amount of flexibility about the specific content expressed in these messages, but it has strong guarantees about a minimal structure for those messages and provides a core vocabulary for doing so.

For the Fedora specification, we were interested in defining a minimal payload format, and so LDN wouldn't really help us define that format, but AS/2.0 will. The important part here is that Fedora isn't defining a custom payload format itself, it is using an existing specification for that payload.

Does that help clarify things?

Aaron

[soro...@gmail.com]

Just to be very explicit: It's perfectly possible to implement LDN and AS/2.0 together, and that would be a fine way to offer messaging for a Fedora implementation as we now understand it.

There is some discussion here:

https://github.com/fcrepo/fcrepo-specification/issues/92

about using RSS for a similar purpose.

---
A. Soroka
The University of Virginia Library

[Ben Cail]

On 04/20/2017 11:13 AM, soro...@gmail.com wrote:
>> On Apr 20, 2017, at 10:50 AM, Esmé Cowles <esco...@ticklefish.org> wrote:
>>
>> Ben C.-
>>
>> I hope that's not where we wind up. I would very much like Hydra to work with several different Fedora implementations (Modeshape, Cavendish, Derby, and hopefully others like Trellis and Static-LDP) and have adopters able to switch between them depending on their needs. And I would hope that Islandora would work with some or all of those same implementations.
>>
>> There may be some clients that have very specific needs that only one Fedora implementation meets, but I hope that's a rare thing and not the norm.
>>
>> -Esmé
>
> I'm not sure why that _would_ happen, unless it turns out that Hydra and Islandora and other sub-communities genuinely have radically different needs, which doesn't seem very likely to me.
>
> What seems far more likely to me is that we would end up with a much bigger "tent" for Fedora implementations and a much bigger selection of implementations from which any given community could choose those that satisfy those needs that _are_ particular to it.
>
> As has become abundantly clear from public remarks by early implementors like Aaron Coburn and Ben Armintor, not every attempt at implementation is going to accomplish or even _try_ to accomplish every piece of the spec _we have already written_. We've already chosen to / had to split off pieces of specification that were once considered core (e.g atomic operations, perhaps fixity as well). I don't think anyone is suggesting that either 1) perfect, thoughtless interchangeability is a realistic goal or that 2) every single site or subcommunity should have a unique implementation of its own.
>
> What I am suggesting is that we must write and share different kinds of specification in the appropriate communities, and that means we cannot ignore the differences between the Fedora community (which has grown by sharing various information architectures) and communities of implementation within it. It's not unimportant for (say) Hydra and Islandora to be able to share implementations. It is more _immediately_ important for Hydra to be able to share implementations within itself (ditto other communities) and what's more, it's a good place to start in order to get to a world where Hydra and Islandora et al. _can_ share implementations. That kind of "bottom-up" growth will let us abstract over real experience to develop interoperability, as opposed to mandating it. That's what leads me to suggest keeping already-built consensus for the whole community in the Fedora spec and keeping assumptions about profiled behavior in some system of profiles (which could evolve much more easily and quickly than the "upper" document and which certainly should feed further revision thereof).

As I understand it, we have a read-only Fedora implementation that fully
conforms to the Fedora specification (Static-LDP). So, any front-end
code that needs to add or update objects can't be written against the
Fedora specification - it needs to be written against a specific
implementation, or some new spec that extends the Fedora spec, or some
informal understanding that it requires read-write Fedora
implementations, not read-only ones.

Am I misunderstanding something? Do most people consider this a non-issue?

-Ben C.

[Esmé Cowles]

Ben-

I think it's not something you'd expect, but something that a lot of specs allow. For example, I've heard of static-file implementations of OAI-PMH, or IIIF, or other specs. I agree that most systems would need to be able to do create/update operations to have a working system, and Static-LDP is only going to fulfill a narrow use case.

So, I'm not worried about it. I would expect the vast majority of implementations to be read-write, but I'm not bothered by a read-only one that fills a gap.

-Esmé

[Benjamin J. Armintor]

Ben - I agree with Esmé, but should probably note that while static-ldp is a fully conformant LDP implementation, that announcement overstates its conformance to the draft Fedora spec (especially for resource management). The question for a client would be whether that matters (ie, does this client actually manage resources). There are well specified mechanisms for detecting that support (HTTP OPTIONS, for example).

I don't think anyone is proposing that useful specifications mean not knowing anything about the implementation - we have lots of RDBMS-driven apps that only have permission to read from the database. But it's worthwhile to talk about interaction patterns when they are available, too!

- Ben

 

> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Graham Triggs]

On 19 April 2017 at 13:32:15, Aaron Coburn (aco...@amherst.edu) wrote:

1. A Fedora server MUST be a conforming LDP server.

5. For any resource that is changed, a Fedora server MUST make a notification available of that change; notifications MUST conform to the ActivityStreams specification.

Forgive me here, but I’ve got a bunch of loosely scanned information floating around my head, and this also relates to something else I’ve been considering. So, a question:

If you are specifying that something must be a conforming LDP server, why would you specify that notifications must conform to the ActivityStreams specification, rather than Linked Data Notifications?

https://www.w3.org/TR/ldn/

Thanks,

Graham

[Aaron Coburn]

Ben,

The static-ldp software serves a very narrow use case -- that of exposing user-managed resources as LDP resources (rather than as just arbitrary web resources). Attempting to modify such resources (PUT, POST, etc) results in a typical LDP-defined response that uses an ldp:constrainedBy Link header. The same information is available via OPTIONS. So an LDP client is able to discover these capabilities (or lack thereof) in a well-defined manner. Which is to say that insofar as Hydra and Islandora clients follow those LDP-defined rules, there would be little to no changes required. In addition to just being an LDP implementation, though, static-ldp also supports RFC 3230 (instance digests) as outlined in the draft Fedora specification (it actually supports instance digests for all resources -- not just LDP-NRs).

Certainly, an institution could put resources on a web server and link to them with Fedora right now:

</some-fedora-resource> ore:aggregates <http://example.com/a-static-resource> .

What static-ldp does is make those resources act as LDP resources along with some of the additional capabilities defined by the Fedora spec.

The other goal of this software was to try to write the very simplest possible software that conforms to the Fedora specification. It was extremely simple to implement, it is simple to deploy and its runtime characteristics are such that it is very fast. But it is certainly not intended to be a "full-featured" repository application.

I suppose that it is conceivable that an institution would use _only_ static-ldp, but that seems extremely unlikely and not one that I would recommend. The basic idea of the software is that it is used in tandem with another R/W Fedora system.

Aaron

[Benjamin J. Armintor]

Aaron-

I definitely did not mean to undersell the project! I think static-ldp makes a valuable intervention and illustrates the flexibility of discoverable capabilities (it also bears pretty directly on some minimal computing efforts here at Columbia).

- Ben

> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Jared Whiklo]

I would say this is a non-issue, mostly because Static-Ldp does _not_
conform to the current Fedora API specification.

While the LDP spec makes PUT/POST/PATCH optional, the Fedora API spec
requires all three of these.

So Static-Ldp is a cool read-only LDP server implementation, but _not_ a
read-only Fedora implementation.

cheers,
jared

--
Jared Whiklo
jwh...@gmail.com
--------------------------------------------------
Everyone wants a bus service to their door, but no one wants a bus
service in their street.

[Aaron Coburn]

To Jared's comment I would add that nowhere on the Static-LDP github site (https://github.com/trellis-ldp/static-ldp) does anyone make the claim that Static-LDP is a conforming Fedora server (in fact, Fedora is not mentioned at all).

Those operations that Static-LDP _does_ support, do happen to conform to the relevant sections of Fedora, but that is the full extent of its "conformance".

Insofar as there _can_ be a read-only Fedora server, yes, it is a read-only Fedora server, but the Fedora spec also mandates support for PUT, POST and PATCH, which seems to explicitly forbid a read-only implementation. Ergo: static-ldp is not a Fedora, it just looks like one on cloudy days from a certain vantage point. Maybe. But not really.

[Robin Lindley]

+1
Robin

Robin Ruggaber


> On Apr 21, 2017, at 6:39 PM, Fleming, Declan <dfle...@ucsd.edu> wrote:
>
> argue

[Declan Fleming]

Hi folks - Andrew pinged the Fedora Leadership group to pay attention to this thread.

 

I'm not going to pretend that I've stayed current on F4/F5 architecture, but I've learned a lot reading what you all have to say about the spec and the process.

 

It is a leadership goal to get the current spec "finished" asap, and we're discussing ways to get there.  I can see how there are a lot of meta-spec discussions happening, and that can be healthy at the start of a process, but seems to be impeding spec completion now.

 

This spec is just a document that we can agree with and/or argue about.  Can we focus on completing what's in front of us now, then commenting on the strengths and flaws of that document?  I'd like to get to v1, then have a more in-depth discussion (preferably NOT as walls of text ;)) in a community focused forum focused on collaboration and making products that are designed to make all of our jobs easier.

Declan

[Benjamin J. Armintor]

Hi Declan,

You're right that there's a lot of conversation about the goals of this spec effort, owing to either a drift in assumptions among the varying contributors or an increasingly explicit lack of agreement about the goals and assumptions. There's also a conversation regarding the number of editors on the spec, process for editing, etc.

For a number of reasons, those disagreements appear to be impeding our arrival on a spec now: We have an unclear scope of goals, distribution of responsibility and mechanisms for conflict resolution. Here's what I would suggest FLG needs to do:

1. Agree on a charter for the spec - what the delivered documents ought to be and supporting software should be (the first deliverable should probably be an editorial process).

2. Identify in plain terms the constituencies/audiences for the effort as the FLG is chartering it.

3. Formally nominate editors (the W3C LDP spec has 3 editors; IIIF has 5). Right now there's 10 different people that have edited the API spec, which seems difficult to justify given the two aforementioned as benchmarks and also makes conflict resolution difficult. It's also the same kind of ad hoc, volunteer effort that drives much of the software development in this space, which the Hydra Partners in the FLG at least should be wary of. I'd suggest that the FLG should nominate 3 employees from the member institutions they represent, or that they can extract a promise of in-kind labor from.

There had been discussions about delivering these documents to the community in May; that seems unlikely. It's important that the FLG recognize in this moment a request of support from their employees contributing to the spec effort to date. 

- Ben

ps: I tried to keep that under the WoT threshold, I promise.  

--

You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

[Jon Stroop]

Just two quick comments:

 

> I'd like to get to v1, then have a more in-depth discussion (preferably NOT as walls of text ;))

 

I don’t want to try to map the IIIF community experience onto the Fedora community too closely, but one lesson learned that I think is relevant, is that there should not be any major or minor releases of a spec until there are two independent working implementations. I’d strongly suggest Fedora follow this rule as well—you really can’t know what assumptions a single implementation is making that are inadvertently driving the spec and 0.9.42-beta is psychologically very different from 1.0.0.

 

Second, closely related, just in case it’s helpful, the IIIF editorial process is a little buried on the site, but it might be worth a look for reference: http://iiif.io/api/annex/notes/editors/

 

-Jon

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.

To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

 

--

You received this message because you are subscribed to the Google Groups "Fedora Tech" group.

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.