API Specification

95 views
Skip to first unread message

Aaron Coburn

unread,
Sep 9, 2017, 5:06:12 PM9/9/17
to <fedora-tech@googlegroups.com>, Fedora Community
Hello,

Typically, issues related to the API specification are raised on the specification github repository, but I would like to bring up a more general issue that, I think, warrants broader community discussion and input. In particular, there are a few places that the specification seems to drift away from standard practices of HTTP servers, and this concerns me both as someone working on implementing a conforming server and as someone who will be interacting with that API through client applications.

Basically, the issue is this: optional headers are described in a such a way that servers which don't support them must provide special error handling. Three examples of this are:

Content-Type: message/external-body (for external body resources)
Depth (for recursive delete operations)
Link: <...>; rel="acl" (for client-location-defined ACL resources)

And I will note that there are good reasons why a server wouldn't support these particular features. But the Fedora specification, as currently written (the third item is part of a proposed change), requires that servers which do not support those behaviors, MUST respond with 4xx errors declaring their non-support.

This is not how any other optional header on any HTTP endpoint that I'm familiar with works. Typically, a server that does not support a particular header simply ignores it: it is as if the header were never provided.

There are lots of examples of this, but I will provide two that people on this list should be very familiar with: content negotiation via "Accept" and representation manipulation via "Prefer". Both of these headers are optional under their respective specifications (HTTP and LDP).

A server that does not support Accept does not need to declare its non-support, rather support for the header is declared pro-actively via a "Vary" header (same with Prefer). If a web resource (e.g. a JPEG image) does not support content negotiation, a request with "Accept: image/png" will simply ignore the header and provide the JPEG. If the resource _does_ support content negotiation, but the request cannot be fulfilled, then a 406 error is returned. A client can easily tell if content negotiation was accepted by inspecting the Content-Type header of the response.

Likewise, with Prefer, support is declared via the Vary header, and if a server was able to handle the request, a special Preference-Applied header is part of the response. Thus, if a server does not support Prefer, it simply ignores the header.

Contrary to this, the Fedora specification currently requires that any server which does not support Content-Type: message/external-body (for external content), or Depth (for recursive deletes) or Link: <...>; rel="acl" (for client-defined ACL resource locations) that those servers MUST provide special handling declaring that they do not support these optional behaviors.

I think this is wrong.

Instead, I think that a server that _does_ support these headers, ought to indicate to a client that those behaviors are supported.

An example of this with the Link: <...>; rel="acl" is simple: if a client provides an ACL location via a link header, if the response contains the location of that ACL resource, the client now has enough information to know if the request was successful. There are surely similar ways for a server to handle the other situations, such as responding with a list of the resources that were deleted via recursion or the use of a Content-Location header in the response for external content.

Either way, the principle is this: a server that does not support an optional behavior should not have to add extra code to _not_ support the behavior.

Regards,

Aaron Coburn







Esmé Cowles

unread,
Sep 10, 2017, 7:03:24 AM9/10/17
to Fedora Tech
In each of these cases, requiring the Fedora implementation to return a 4xx error is a compromise — the main alternative would be to simply require that the feature be supported.

Of course, sometimes requests fail for various reasons, and we certainly acknowledge that an implementation doesn't need to support every possible value for those headers. For example, with Depth, a server like Fedora 4.7 that only supports recursive deletes would not be able to do a non-recursive delete if a Depth: 0 header were included. This is a great example of why failing the request is the right thing to do: ignoring the Depth header could result in deleting many more resources than the client intended.

If an implementer is determined not to support a feature, they can take advantage of this and just return a 4xx response for any request with the header. I personally think that's not a good implementation choice, because these are all features that are being used by production applications based on Fedora 4.7, and the API specification is merely formalizing how clients and servers should coordinate them in ways that are better aligned with the HTTP, LDP, and other specifications.

-Esmé
> --
> You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

acoburn

unread,
Sep 10, 2017, 7:58:56 AM9/10/17
to Fedora Tech
While I appreciate the perspective of the editorial board on this issue, the reason I bring this up here is specifically to bring in perspectives from those who are not editors. Again, my concern here has nothing to do with the potential usefulness of these particular behaviors, rather it has to do with the underlying interaction pattern that departs from most (all?) typical HTTP server interactions with optional header values. I have argued exhaustively elsewhere why these behaviors should not be required of a Fedora server, and I do not see that as a possible alternative.

What I am suggesting is that the underlying design of these header interactions needs to be reconsidered by the editors. In its current form, it is problematic for the reasons described earlier.

Aaron

Joshua Westgard

unread,
Sep 10, 2017, 9:22:51 AM9/10/17
to Fedora Tech
This is a tough one, as I can see the merits of both approaches.  A couple initial reactions:

(1) While adhering to typical and best practices from the world of HTTP/LDP is a good thing, I'm OK with seeing Fedora as a more LAM-domain-specific extension of those practices and therefore with having certain places where it differs from what is typical.  Whether in this specific case the different approach is the best one, I'm not sure.

(2) Would it be possible to do the alternative that Esmé mentioned (namely require all these features to be supported for full compliance), and just live with a community where some servers implement only part of the specification?  In some ways this would seem to have the same effect on clients -- i.e. you'd have to have a method of determining which features are supported by a particular server. And in turn maybe this is what the current compromise provides (i.e. a common and defined way of responding to un-implemented requests). 

Josh

acoburn

unread,
Sep 12, 2017, 10:20:39 AM9/12/17
to Fedora Tech
I think Josh's comment really gets at the crux of this issue, namely: is Fedora's vision for the future expansive or is it domain specific?

If the vision for Fedora's future is domain specific, especially if its goal is to be a specialized back-end for Islandora and Samvera software, then adding restrictions and specializations would be a good idea. Alternately, if Fedora's vision is that it provides a fundamental piece of data infrastructure for the read-write web, then I would argue that adhering closely to standard practice would be paramount.

Some guidance from Fedora-Leaders would be useful here.

Thanks,
Aaron

Benjamin Armintor

unread,
Sep 12, 2017, 1:47:14 PM9/12/17
to fedor...@googlegroups.com
Aaron,

I'm going to respect your interest in non-editor opinions on the matter, but citing the disjoint between several mutating behaviors in the spec and the behavior of GET with Accept headers is a somehwat misleading comparison. Can you identify some similar examples of header disregard on writeable REST APIs?

I'd also like to note that the Accept-Post behavior you cite for external content is not a departure from LDP, but a strengthening of the requirement in OPTIONS. Are you advocating leaving that requirement as a SHOULD?

Regards,

Ben

To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

Aaron Coburn

unread,
Sep 12, 2017, 2:19:47 PM9/12/17
to fedor...@googlegroups.com

> Can you identify some similar examples of header disregard on writeable REST APIs?

Sure, here are two off the top of my head:

Slug. Support for this is MAY in HTTP, and even if a server supports Slug headers, a server is under no obligation to honor the client's request.

Upgrade. Not all servers support the Upgrade mechanism, and even if a server does support it, there's this from the HTTP 7230 spec:

A server MAY ignore a received Upgrade
header field if it wishes to continue using the current protocol on
that connection.


> I'd also like to note that the Accept-Post behavior you cite for external content is not a departure from LDP, but a strengthening of the requirement in OPTIONS. Are you advocating leaving that requirement as a SHOULD?

You misunderstand me in this regard. The Accept-Post mechanism is precisely a positive example of what I am suggesting: it indicates pro-actively to clients that the origin server supports external content. My complaint about the spec is that, even when a server does not indicate such support and a client goes ahead and tries to send "external content", the server is required (according to the Fedora spec) to respond with a 4xx error. Instead, what I am suggesting there is that the spec be silent on what a non-supporting server does, and rather requires that a server that supports external content respond with some header indicating that the request was honored.
> You received this message because you are subscribed to a topic in the Google Groups "Fedora Tech" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/fedora-tech/VLDQOwhhTqY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to fedora-tech...@googlegroups.com.

Mike Giarlo (Google Groups)

unread,
Sep 12, 2017, 5:08:20 PM9/12/17
to fedor...@googlegroups.com
All good points. It's not clear to me that a few MUSTs in the specification mean Fedora is a poor fit for read-write web infrastructure. For now, I'm in favor of entrusting the editors to finish work on the first version of the Fedora specification, at which point we can iterate based on early implementation experience and revise the specification as needed.

FWIW, I would also support deciding to make these features required of Fedora implementations if others in the community feel strongly about the 4xx issue.

Thanks, all.

--
Michael J. Giarlo
Technical Manager, Hydra-in-a-Box project
Software Architect, Digital Library Systems & Services
Stanford University Libraries

________________________________________
From: fedor...@googlegroups.com <fedor...@googlegroups.com> on behalf of Aaron Coburn <aco...@amherst.edu>
Sent: Tuesday, September 12, 2017 11:19
To: fedor...@googlegroups.com
Subject: Re: [fedora-tech] API Specification

Aaron Coburn

unread,
Sep 12, 2017, 6:51:13 PM9/12/17
to fedor...@googlegroups.com
Mike, all of the points I am making here (and have made throughout this API spec process) are based on an early implementation -- an implementation that is nearing completion. These are not theoretical arguments; they are very practical. And as someone who is attempting to implement these specifications, I am finding problems and raising them publicly.

It seems, however, that the general sentiment is not in my favor, so I will simply rest my argument on the point and spend my time focused on the underlying specifications: LDP, Memento, WebAC, Instance digests and Activity Streams.

Thanks all for the input.

Aaron

Joshua Westgard

unread,
Sep 13, 2017, 11:49:54 AM9/13/17
to Fedora Tech
I've been around Fedora long enough to know that if Aaron C. says "here be dragons" you would be wise to watch out.

But at the same time I think we do have a more narrow and specialized focus with this API specification than the whole of the read-write web community.  Or perhaps better to say, the interests and goals (and pragmatic choices?) of the LAM community as embodied in the spec -- while they could certainly benefit many constituencies of the wider web -- are never going to be the best fit for every use case within the wider web.  The first thing we have to do to see this work gets adopted more widely, however, is to produce something usable within this more narrow community.

We are implementing a repository ecosystem with Fedora as a first-class citizen (sorry about the mixed metaphor).  We haven't implemented either Islandora or Samvera on top of our fcrepo at this point, though we do use Blacklight.  We will likely be adopting additional pieces of the Samvera framework in the future because we know how the story will end if we try to build it all ourselves and it isn't pretty.  But for us Fedora will never just be the backend of whatever application we're using -- it represents an approach to sustainable long-term web infrastructure that is central to what we are trying to achieve as a division within the Libraries.

Josh

Jared Whiklo

unread,
Sep 13, 2017, 1:08:27 PM9/13/17
to fedor...@googlegroups.com
The theory of the API (to my understanding) was that others could build
a Fedora with features that might be specific to their use cases and
those use cases would be from as wide a community as possible to allow
growing the community.

I am not implementing the Fedora API but I can see it as odd that if a
header is optional, then I'd still have to check for them and halt if
something I don't support and is optional is requested.

If the header really is optional, then it is up to the client to use the
appropriate Fedora for their use case. Presumably the Fedora they choose
would support that header and its interaction.

Either that or make those headers required in the API, which (again I
feel) will cause the breadth of different implementations to be reduced
or narrowed.

cheers,
jared

On 2017-09-13 10:49 AM, Joshua Westgard wrote:
> I've been around Fedora long enough to know that if Aaron C. says "here
> be dragons" you would be wise to watch out.
>
> But at the same time I think we do have a more narrow and specialized
> focus with this API specification than the whole of the read-write web
> community.  Or perhaps better to say, the interests and goals (and
> pragmatic choices?) of the LAM community as embodied in the spec --
> while they could certainly benefit many constituencies of the wider web
> -- are never going to be the best fit for every use case within the
> wider web.  The first thing we have to do to see this work gets adopted
> more widely, however, is to produce something usable within this more
> narrow community.
>
> We are implementing a repository ecosystem with Fedora as a first-class
> citizen (sorry about the mixed metaphor).  We haven't implemented either
> Islandora or Samvera on top of our fcrepo at this point, though we do
> use Blacklight.  We will likely be adopting additional pieces of the
> Samvera framework in the future because we know how the story will end
> if we try to build it all ourselves and it isn't pretty.  But for us
> Fedora will never just be the backend of whatever application we're
> using -- it represents an approach to sustainable long-term web
> infrastructure that is central to what we are trying to achieve as a
> division within the Libraries.
>
> Josh
>
>
>
> > ________________________________________
> > From: fedor...@googlegroups.com <javascript:>
> <fedor...@googlegroups.com <javascript:>> on behalf of Aaron Coburn
> <aco...@amherst.edu <javascript:>>
> > Sent: Tuesday, September 12, 2017 11:19
> > To: fedor...@googlegroups.com <javascript:>
> <https://groups.google.com/group/fedora-tech>.
> >>> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the
> Google Groups "Fedora Tech" group.
> >> To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com <javascript:>.
> >> To post to this group, send email to fedor...@googlegroups.com
> <javascript:>.
> <https://groups.google.com/group/fedora-tech>.
> >> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
> >>
> >>
> >> --
> >> You received this message because you are subscribed to a topic
> in the Google Groups "Fedora Tech" group.
> >> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/fedora-tech/VLDQOwhhTqY/unsubscribe <https://groups.google.com/d/topic/fedora-tech/VLDQOwhhTqY/unsubscribe>.
>
> >> To unsubscribe from this group and all its topics, send an email
> to fedora-tech...@googlegroups.com <javascript:>.
> >> To post to this group, send email to fedor...@googlegroups.com
> <javascript:>.
> <https://groups.google.com/group/fedora-tech>.
> >> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to fedora-tech...@googlegroups.com <javascript:>.
> > To post to this group, send email to fedor...@googlegroups.com
> <javascript:>.
> <https://groups.google.com/group/fedora-tech>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
> >
> > --
> > You received this message because you are subscribed to a topic in
> the Google Groups "Fedora Tech" group.
> > To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/fedora-tech/VLDQOwhhTqY/unsubscribe <https://groups.google.com/d/topic/fedora-tech/VLDQOwhhTqY/unsubscribe>.
>
> > To unsubscribe from this group and all its topics, send an email
> to fedora-tech...@googlegroups.com <javascript:>.
> > To post to this group, send email to fedor...@googlegroups.com
> <javascript:>.
> <https://groups.google.com/group/fedora-tech>.
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fedora-tech...@googlegroups.com
> <mailto:fedora-tech...@googlegroups.com>.
> To post to this group, send email to fedor...@googlegroups.com
> <mailto:fedor...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
Jared Whiklo
jwh...@gmail.com
--------------------------------------------------
Comedy's a dead art form. Now tragedy, ha ha ha, that's funny. -- Bender

signature.asc

Esmé Cowles

unread,
Sep 13, 2017, 1:34:39 PM9/13/17
to fedor...@googlegroups.com
Jared-

I'd just like to say that two of the features being discussed aren't optional. External content and using the Depth header to indicate what kind of deletion behavior is desired are required features.

But not every permutation of values is required, and there are some values that are just plain invalid (Depth: -1, external content with access-type="carrier-pigeon"). So the spec says that you should return error status codes for the values you don't support, instead of creating incorrect content (or deleting extra content). So I think it's more accurate to say that supporting those headers is required, but supporting every possible value in the headers is not.

As an accommodation to implementers who didn't want to support those features, we agreed that in practice an implementation could simply return an error for all of the values. It's not ideal, but at least a client would know when its request wasn't acceptable, and could then fall back on some other approach.

We haven't gone down the road of specifying which values must be supported, in part because we knew it would lead to conflict that would be harder to resolve. But IMHO, it would be an improvement in the spec to say, e.g., that external content with access-type="URL" must be supported. Or you must at least support either Depth 0 or infinity.

-Esmé

Jared Whiklo

unread,
Sep 13, 2017, 4:09:23 PM9/13/17
to fedor...@googlegroups.com
Thanks Esme,

Sooooo clearly never mind my comments.

Support for the header is required, but support for any sort of
interaction based on the header is not. That...yeah its something to
consider.

I got nothing.

cheers,
jared
When everything's coming your way, you're in the wrong lane and going
the wrong way.

signature.asc

Benjamin Armintor

unread,
Sep 26, 2017, 10:56:34 AM9/26/17
to fedor...@googlegroups.com
Pursuant to Esmé's point about requirement for Depth, I've tried to outline an alternative approach without Depth at all - this similar to implicitly requiring Depth: infinity, or Depth: 0 if DELETE support is not advertised. This seems like a viable alternative to elaborating the requirements, at least worth discussing.

I've described the possible approach and will put together a PR for discussion at https://github.com/fcrepo/fcrepo-specification/issues/234

- Ben


>> To post to this group, send email to fedor...@googlegroups.com

>> Visit this group at https://groups.google.com/group/fedora-tech.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> Jared Whiklo
> jwh...@gmail.com
> --------------------------------------------------
> Comedy's a dead art form. Now tragedy, ha ha ha, that's funny. -- Bender
>
> --
> You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.

> To post to this group, send email to fedor...@googlegroups.com.
> Visit this group at https://groups.google.com/group/fedora-tech.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages