[API-X] Initial workflow graphs

[Stefano Cossu]

Hello API-X Group,
Following up the latest meeting, I have drafted some swim lane diagrams
illustrating example usages of the API Extension Architecture.

Find attached files.

I would like some feedback before I go ahead with further use
cases. I may also need some clarification about some use cases that I
did not propose myself and may need to understand better.

Adam S. and I had a brief discussion on the use cases that would best
cover a wide range of workflows. The list came down to:
- Validation [1] (provided here are both synchronous and asynchronous
workflow - the latter applicable to a variety of similar uses)
- Provenance stream [2] (also attached here)
- Representation from a graph of objects [3] (TODO)
- Resources as MODS/XML [4] (TODO) - may this be merged with the
previous one in a more general use case of I/O manipulation?
- Extra-repo access control [5] (TODO)

It may be worth noting that the API-X layer is rather lean, with the
extensions not containing any code at all. This approach relies mostly
on external services to provide custom code to handle any kind of
filtering, manipulation and validation of request and response data.

Pending is also an update of the component graph according to the last
meeting discussion. I also think it would be helpful to have a general
workflow graph illustrating more in detail the pre- and post-extension
operations common to all the above scenarios (service discovery,
extension prioritization, queuing, etc.) which I left out in these
graphs for the sake of brevity.

Feedback is appreciated!

Thanks,
Stefano

[1]
https://wiki.duraspace.org/display/FF/AIC+Use+Case%3A+Content+and+Structural+Validation
[2]
https://wiki.duraspace.org/display/FF/JHU+API-X+Use+Case%3A+Provenance+stream
[3]
https://wiki.duraspace.org/display/FF/SI+-+One+Representation+Derived+from+a+Graph+of+Objects
[4] https://wiki.duraspace.org/pages/viewpage.action?pageId=70583487
[5]
https://wiki.duraspace.org/display/FF/AIC+User+Case%3A+Extra-Repository+Access+Control

--
Stefano Cossu
Director of Application Services, Collections

The Art Institute of Chicago
116 S. Michigan Avenue
Chicago, IL 60603

[Yinlin]

Hi Stefano,

Thanks for working on the diagrams. Just a minor suggestion,  in both asynchronous and synchronous diagrams, the "Ingest resource" and "Forward request to destination" can move below the "Validation pass?" box. "Persist resource" and "Send response" boxes also move down. Thus, the that "YES" arrow can change direction to downward and the diagram would be more coherent.

Just my 2 cents.
 
- Yinlin

[Stefano Cossu]

Yinlin,
You are absolutely right. It makes more sense visually. Find the
updated graphs attached.

I also have used the label "Forward request to destination" in the
API-X core lane and added the bypassing validation request to be
consistent in both graphs.

Thanks,
Stefano

[Elliot Metsger]

Hi Stefano,

Thanks for these diagrams!  Have a couple of comments.

On Wed, Feb 3, 2016 at 5:34 PM, Stefano Cossu <sco...@artic.edu> wrote:

<snip>

 

- Provenance stream [2] (also attached here)

These comments are addressed to the Provenance stream example, but may apply to the other examples as well.  I'm curious what folks think on these:

1. In the Prov. Extension column, the Extension is receiving the request, and evaluating whether or not it applies to itself.  I am wondering about the general pattern, and responsibilities. Is the API-X core going to hand off the request to each registered extension for evaluation?  Or should the API-X core have enough information from the proposed Service Discovery and Binding service/registry to make that evaluation on its own?

2. If the "Provenance Extension" column is renamed to "Service Discovery & Binding" then (IMO) the responsibilities are more clear, and the individual boxes in that column are a better fit (though the responsibility of evaluation of a request as it applies to an extension might be better executed in the API-X core, consulting the SD&B as needed),

3. If the conditions for the request do not hold, is the proper behavior to forward to the destination?  For example, what happens if a request identifies a service resource that hasn't been advertised?  E.g. a request for /path/to/resource/svc:prov when "svc:prov" hasn't been advertised and is actually not there.

 

- Representation from a graph of objects [3] (TODO)
- Resources as MODS/XML [4] (TODO) - may this be merged with the
  previous one in a more general use case of I/O manipulation?
- Extra-repo access control [5] (TODO)

I think the feedback from the existing diagrams will help inform the remaining diagrams, so leaving these as TODOs is fine with me.

 

It may be worth noting that the API-X layer is rather lean, with the
extensions not containing any code at all. This approach relies mostly
on external services to provide custom code to handle any kind of
filtering, manipulation and validation of request and response data.

Yes, exactly!  In the provenance example, there may not be any "Extension" to speak of at all, if the API-X core is responsible for determining whether or not a particular Extension applies to the request.  The API-X core determines that the request is a provenance service request by consulting the SD&B registry, and forwards it on.

<snip>

[Aaron Birkland]

Thanks Stefano,

I updated the meeting page to link to your latest images:

https://wiki.duraspace.org/x/mohcB

I think we should discuss the 'validation pass' stage (under the API-X core column) and generalize it.  For the synchronous use case, the extension is essentially being used as a filter for requests to and from the repository.   So the API-X core (without knowing what validation is or does) would need to somehow know whether to continue routing the request to its destination, or return an error response.  The error response might be better off originating from the extension, not API-X.  That way the extension could craft an appropriate error response (say something like "validation failed, and here's why" in the response, and choose a reasonable HTTP response code).  

For the async case, could the extension return any kind of response that would indicate the status of validation?  Say, a link to a resource that will contain the results of a validation run on a given object?

  -Aaron

On Fri, Feb 5, 2016 at 12:04 PM, Stefano Cossu <sco...@artic.edu> wrote:

Yinlin,
You are absolutely right. It makes more sense visually. Find the
updated graphs attached.

I also have used the label "Forward request to destination" in the
API-X core lane and added the bypassing validation request to be
consistent in both graphs.

Thanks,
Stefano

On Fri, 5 Feb 2016 07:18:12 -0800 (PST)
Yinlin <ylc...@vt.edu> wrote:

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To post to this group, send email to fedor...@googlegroups.com.
Visit this group at https://groups.google.com/group/fedora-tech.
For more options, visit https://groups.google.com/d/optout.

[A. Soroka]

If the extension is returning a “report” or description of how the resource does or does not meet validation, it becomes a kind of variation on the general theme of resource transformation. Is it possible that clever architecture in API-X could make validation available as two interacting components of transformation? That might be a nice “factoring”.

---
A. Soroka
The University of Virginia Library

[Aaron Birkland]

If the extension is returning a “report” or description of how the resource does or does not meet validation, it becomes a kind of variation on the general theme of resource transformation. Is it possible that clever architecture in API-X could make validation available as two interacting components of transformation? That might be a nice “factoring”.

Yes, I like the idea of viewing validation as variation of general theme of resource transformation. 

  -Aaron