Core API

[Tom Christie]

Hi folks,

Now seems like a good time to introduce a system I've been considering for a while, and have recently started working on in earnest.

Core API - https://github.com/core-api/core-api

Currently there's a specification, a python client library implementation, and an example service for testing purposes.

Couple of things that I think are particularly interesting here:

* Core API fully separates the document layer (which is what the client interacts with) from the encoding and transport layers.

  This is something I think it does really well. You'll notice when reading through the spec that there's no mention whatsoever of the underlying encoding. The initial default JSON-based encoding *has* all been fully designed, and is expressed fully in the Python client library, but I've not yet documented it. That, and the transport specification will be coming in the next few days.

* Core API allows composability of components. Eg a transition may effect only part of the document tree, but return a complete new document once it's taken effect.

The naming of the transition types is potentially an open design decision. It'd also be interesting to discuss if anyone thinks there are transition types that are missing and required.

Having had a quick scan through the history of this list the "Representor" pattern seems to be pretty much the same as what I've arrived at with the "Document layer" in Core API. (?)  It was really interesting how nicely the clean separation mapped onto the client library implementation. I was able to first develop and test the complete Document interface completely in isolation, without ever having to touch anything to do with encoding or network requests.

Anyways, I won't ramble on for now - I'm intend to be very actively pursuing the project. There's a whole long list of things at the moment I'm considering for future work. And it would be great to hear what you folks think of the design to date.

Feel free to have a play around with the python client library using the 'http://coreapi.herokuapp.com/' example service. The only outstanding bit of work for the python client library is to implement the error message handling. Everything else in the spec is handled, and the library has great coverage, so should already be pretty solid by this point.

Thanks all,

  Tom

Aside: The precursor to this work was 'DocJSON' - which I never pursued further as I realized that I hadn't quite gotten the composability aspect properly figured out at the time (you'd always have to return the complete root document from the server after each transition). DocJSON also never made the leap to separating the document model/specification from the encoding model/specification.

[Tom Christie]

I've now published the full document, encoding and transport specifications for Core API:

http://core-api.github.io/core-api/

Again, I'd be super interested in any feedback from folks here.

Particularly interested in any thoughts on the naming of transition types, the relationship between the 'document layer' and the 'representor pattern' that's been discussed, and the composability aspect of documents.

Thanks & have a great weekend.

  Tom

[Joost Cassee]

Hi Tom,

Thanks for the link. I have looked over the documentation, but I find
it difficult to get a feel for what the JSON would look like. I always
go for the examples first, and in this case they are Python snippets.
Do you have some real "over the wire" examples?

Regards,
Joost

> --
> You received this message because you are subscribed to the Google Groups
> "Hypermedia Web" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hypermedia-we...@googlegroups.com.
> To post to this group, send email to hyperme...@googlegroups.com.
> Visit this group at http://groups.google.com/group/hypermedia-web.
> For more options, visit https://groups.google.com/d/optout.



--
Joost Cassee
https://jcassee.com

[Tom Christie]

Hi Joost,

> I find it difficult to get a feel for what the JSON would look like

I'll do some re-jigging to make that more prominent, but you can see an example on the encodings doc...

http://core-api.github.io/core-api/specification/encoding/

It'll probably be worth me revisting the index page to give a better idea of what network requests/reponses are occurring at each point of interaction.

Cheers,

  Tom

[Joost Cassee]

Thanks! I'm not sure if you have experience with either HAL or JSON
API, but if so, could you say something about what makes Core API
different?

Regards,
Joost

[Tom Christie]

Sure. I should probably take some of this into a proper "Motivation" section in the docs, but briefly.

* Core API has a separation of the document layer from the encoding layer. The document layer is the point at which the client interacts with the service. That's different because you can actually support multiple encodings, without affecting the client interaction at all. This looks to me like the 'representor' pattern that's been mentioned elsewhere on the list, but I don't otherwise think I've seen similar stuff. (I probably don't do a good enough job of watching what else is going on tho, so don't take my word for it)

* Core API is general purpose supporting links and actions either of which may be parameterized. That's not particularly radical/new, but it is currently pretty well featured, and will be more so once the specification also includes other media upload and download as well. There's a range of different levels of support for that across the various hypermedia formats, with eg HAL just allowing 'safe' linking, and JSON API being more of a data exchange/sync model (I think?) Apologies to Authors and contributors if I've mischaracterized either of those.

* Core API supports document composition, and only updates the relevant part of the document tree when undertaking transitions. I think the way it does that (without simply being sync this part fo the document) is broadly new. (?)

Apologies to all if I'm saying anything incorrect here :)

[Tom Christie]

So, Core API now has both JSON and HTML encodings specified.

The end result of this is that you can now interact with a Core API interface either directly through a web browser, or programmatically.

There's a live example here: http://coreapi.herokuapp.com/

The specification is at: http://www.coreapi.org/

I'd still be very interested in any feedback, specifically around:

* Are there other hypermedia approaches I should be looking at that separate the DOM specification from the encoding specification?

* Are there similar approaches to composability, that update only the required part of the document on each transition.

* Other hypermedia formats that have an HTML encoding specified, and can be directly interacted with in the browser, without the indirection of an API browser?

* What parallels do the transition types I've identified for Core API have with other approaches?

I'm also very mindful at the moment of trying to find the right language to help explain to folks why I think the separation of document layer and encoding/transport layers is important, so any thoughts on that also most welcome.

Thanks for your time, all.

Cheers,

  Tom

[Mike Drawback]

Hi Tom,

I followed DocJSON closely, so I was very happy to see a new project come along all of a sudden to take its place!

I'm wondering how the composability aspect will limit the ability to create richer interfaces. For example, you had "tabs" in the DocJSON example project that could filter the notes for complete or incomplete. If the CoreAPI example had a complete=false tab and marking a child note as completed only updates that child document, wouldn't the client still display it? How would you handle that in the new example project?

Please forgive me if I am not understanding transitions correctly or overlooking a simple solution. I guess one advantage of having to return the entire root document with each transition was that the whole application state was controlled by the server, so any part of a complex document could react to the user's actions.

Thank you for all your work on this, it is very interesting!

[Tom Christie]

Thanks Mike!

> I'm wondering how the composability aspect will limit the ability to create richer interfaces. 

Good point - I hadn't *fully* appreciated some of the subtleties there.

One outstanding issue to be resolved is if the link transition types enforce the final transition, or if they only indicate the expected transition.

If we choose the former then we can have links in composable documents that may end up either update *or* delete the document. Let's say we have a list of notes filtered by 'foo', and we then update one of those to no longer include the text 'foo'. On the wire the update would look like:

    PUT /notes/abc123/?search=foo

    <some data>

    204 NO CONTENT

The empty document here would effect a removal rather than a replacement of the child document.

Another aspect of the point you raise is that Core API *can* return new top level documents (as per the DocJSON style) if needed. The action transition type allows this. There will probably also be an 'iaction' transition type. I wasn't aware of a good use case for the 'iaction' type until you raised this point, but along with 'refresh' I believe it'd fill the last remaining valid transition types. (Ie. I don't believe any more valid combinations in the table of behavior)

Having said that - returning top level documents with each transition doesn't feel like good style, and was one of the motivators on the DocJSON evolution into Core API. (The proper separation of the document layer, and better understanding of the difference between transition types are others)

> Thank you for all your work on this, it is very interesting!

Indeed - I'm looking forward to building the javascript client for this as the next step. The ability to create more meaningful interface layer, the benefits of a shared ecosystem of client libraries, and the browsable aspects of Core API all seem like big potential wins to me.

I'm mindful that the document interface imposes a different set of constraints on how you build the service interfaces, and I've been doing what I can to compare this to other historical approaches. One thing that I find interesting in that comparison is that the python client library (as a reference point) is not particularly pythonic. Documents in that implementation are immutable objects, and transitions are not effected in the same way as python method calls. I think that's telling in that it demonstrates a document-focused approach, rather than a language-focussed approach.

Anyways, thanks again for your time and feedback.

  Tom

[Tom Christie]

Hi folks.

I've been putting in a bunch of work to Core API that some of you might find interesting.

Firstly I've created a second example service. This is a trivial treasure hunt game - there's a 3x3 boards, and you've got 5 attempts to guess which square the treasure is in. The intention is to demonstrate a no-CRUD type application, in addition to the existing notes example.

Secondly, there's now a command line client, for interacting with Core API services.

I think the command line client is a really nice example of how hypermedia based services can be interacted with in a more meaningful way than ad-hoc RESTish APIs.

If you're interested and want to try the command line client, then:

* Make sure you've got python installed.

* Install the Core API command line client using `pip install coreapi`

* Fetch one of the example services using either `coreapi get http://notes.coreapi.org` or `coreapi get http://game.coreapi.org`

* Interact with the service using the `coreapi action ...` command.

See the documentation for more details.

As previously noted you can also interact with either example service directly from your browser, which will receive an HTML rendering of the document rather than the JSON-based rendering. Simply visit the either http://notes.coreapi.org or http://game.coreapi.org to give that a try.

I've got plenty more upcoming work for Core API planned, which I'll be sharing with this list in due course.

All the best,

  Tom :)

[mca]

good to see this project continues to acquire new features.

one that i'd really like to see is support for additional registered formats (HAL, Cj, Siren, UBER, Atom, etc.). HTML + a proprietary JSON model will limit adoption, IMO.

a key principle in hypermedia-style implementation is to treat the output as a _representation_ of the data, not a _copy_ of it. that allows for new format support in the future as tastes/needs change over time. 

as a small example, i'd love to see the game example add support for the reg'd Maze+XML format.

mca

Mike Amundsen

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

--

You received this message because you are subscribed to the Google Groups "Hypermedia Web" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hypermedia-we...@googlegroups.com.
To post to this group, send email to hyperme...@googlegroups.com.

Visit this group at https://groups.google.com/group/hypermedia-web.

[Tom Christie]

> a key principle in hypermedia-style implementation is to treat the output as a _representation_ of the data, not a _copy_ of it. that allows for new format support in the future as tastes/needs change over time. 

Indeed. CoreAPI does explictly draw that separation by specifying the document layer as distinct to the encoding layer.

Any difficulties in supporting various other formats come down to the differing underlying capabilities and assumptions each make, rather than differences in encodings.

* What primitive datatypes are supported.

* What kinds of transitions are supported.

* What additional characteristics/properties do link and/or resource primitives support.

* Are there constraints on the layout/namespacing/identification of links and/or nested resources within a parent document.

The existence of the JSON-based encoding is down to the existing encodings not quite fully expressing the underlying model I had in mind with CoreAPI.

Having said that, you're clearly completely right about adoption, which has provoked some good thoughts on how I can make the document model a little more well suited to expressing a general-purpose Data+Links+Resources model (Specifically: CoreAPI's link transitions ought to separate out the 'action' from the 'transition', rather than having a single property that identifies both.) and supporting alternative hypermedia encodings.

Adding support HAL and/or Siren would be a good target, and I've got a rough idea how they would map to the document model.

The main awkwardnesses are likely to be:

* HAL's lack of form support or action indicators on links means it wouldn't be able to meaningful support either of the two CoreAPI example services.

* HAL+Siren's "Links and/or nested resources are always top-level elements" vs CoreAPI's "Links and/or nested resources may be contained anywhere within the parent resource"

* Siren's links and/or embedded resources are not named, but are instead identified by their 'rel' and/or 'class' values, in contrast to CoreAPI's links/resources being identified by their position in the document.

The hypermedia project have resolved similar issues by defining their "representor" very broadly, in an attempt to fully encompass a wide range of existing media types. Ideally I'd like to do something similar, but smaller and more constrained, for CoreAPI's "document model".

Cheers,

  Tom

[mca]

Tom:

yep - one of the challenges of supporting multiple formats is getting used to less than 100% fidelity.  happens w/ HTML all the time and, to move ahead, we'll need to get used to that for other formats, too.

hypermedia project is a good effort. there is another (https://github.com/ubiquitary) similar effort going on, too. I've also begun very early work on a server-side only approach and will "out" that in the next couple months. 

anyway, looking forward to seeing more on your project

Cheers.

mca

Mike Amundsen

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

--

[Tom Christie]

I've been spending a lot of time finessing the Core API spec & client library.

Result is that it now has HAL support, so...

* The command line client and python client libraries now support interacting with HAL services.

* The online examples now support Core JSON, HAL and HTML representations. (eg try `curl -H "Accept: application/hal+json" notes.coreapi.org`)

There's a little bit of documentation on the provisos/differences when using HAL.

I'm going to continue putting a bunch of work into this, so expect to see other formats and languages supported in due course.

Some open design question I've got in mind:

* Is 'document' appropriate, or should I consider 'resource' everywhere, while the design is still in the early stages.

* Should the 'inplace=true/false' concept be something more generic (eg transform='inplace/follow') in order to possibly support other transform types in the future (eg pagination controls that append or prepend the return results)

We've been taking some first experimental steps in using Core API at work, and the notion of in-place transforms in particular has really changed how we've been designing our APIs. I've also got plans in place for handling Mike's comment from several months back on how in-place transformations interact with the document as a whole in non-trivial cases.

As ever, any thoughts/feedback on any of this ongoing work are very much welcome!

I've included an example of interacting with a HAL API below, to give a feel for using a Core API client library with HAL.

$ pip install coreapi

[...]

Successfully installed coreapi-1.11.4

$ coreapi get https://api-sandbox.foxycart.com

<Error>

    errors: [

        {

            logref: "id-1453305072"

            message: "The FOXY-API-VERSION request header is required in order to use the API. Please include the following header with all requests: FOXY-API-VERSION: 1"

        }

    ]

    total: 1

$ coreapi headers add FOXY-API-VERSION 1  # Add a custom header to all outgoing requests.

Added header

Foxy-Api-Version: 1

$ coreapi get https://api-sandbox.foxycart.com

<Your API starting point. "https://api-sandbox.foxycart.com/">

    message: "Welcome to the FoxyCart API! Our hope is to be as self-documenting and RESTful as possible. Please let us know if you have any questions by posting in our forum http://forum.foxycart.com or emailing us at help...@foxycart.com. As a last resort, you could read the documentation at http://wiki.foxycart.com. Your first action should be to create an OAuth Client, then a user, followed by a store."

    create_client()

    property_helpers()

    rels()

    token()

$ coreapi bookmarks add foxycart  # Let's add a bookmark so it's easier to get back to.

Added bookmark

foxycart

$ coreapi action create_client  # Try to follow this link.

<Error>

    errors: [

        {

            logref: "id-1453305175"

            message: "No route found for \"GET /clients\": Method Not Allowed (Allow: OPTIONS, POST)"

        }

    ]

    total: 1

$ coreapi action create_client --action post  # HAL doesn't include the action information, so we specify it explicitly.

<Error>

    errors: [

        {

            logref: "id-1453305183"

            message: "redirect_uri can't be blank, redirect_uri is too short (minimum is 1 characters)"

        },

        {

            logref: "id-1453305183"

            message: "project_name can't be blank, project_name is too short (minimum is 1 characters)"

        },

        {

            logref: "id-1453305183"

            message: "company_name can't be blank, company_name is too short (minimum is 1 characters)"

        },

        {

            logref: "id-1453305183"

            message: "contact_name can't be blank, contact_name is too short (minimum is 1 characters)"

        },

        {

            logref: "id-1453305183"

            message: "contact_email can't be blank, contact_email is too short (minimum is 1 characters), contact_email is not a valid email address."

        },

        {

            logref: "id-1453305183"

            message: "contact_phone can't be blank, contact_phone is too short (minimum is 1 characters)"

        }

    ]

    total: 6

$ coreapi action create_client -a post -p redirect_uri=http://example.com/redirect -p project_name=example -p company_name=example -p contact_name=tom -p contact_email=t...@example.com -p contact_phone=123  # Specify several parameters. Use --param for more verbose style.

<This Client "https://api-sandbox.foxycart.com/clients/3731">

    message: "client 3731 created successfully."

    token: {

        access_token: "***"

        expires_in: 7200

        refresh_token: "***"

        scope: "client_full_access"

        token_type: "Bearer"

    }

    attributes()

$ coreapi credentials add api-sandbox.foxycart.com "Bearer ***"  # Add an Authorization header, for all requests to this domain.

Added credentials

api-sandbox.foxycart.com "Bearer ***"

$ coreapi bookmarks get foxycart  # Go back to our bookmarked URL.

<Your API starting point. "https://api-sandbox.foxycart.com/">

    message: "Welcome to the FoxyCart API! Our hope is to be as self-documenting and RESTful as possible. Please let us know if you have any questions by posting in our forum http://forum.foxycart.com or emailing us at help...@foxycart.com. As a last resort, you could read the documentation at http://wiki.foxycart.com. Your first action should be to create an OAuth Client, then a user, followed by a store."

    client()

    create_user()

    property_helpers()

    rels()

    reporting()

    token()

[Tom Christie]

Yikes, the formatting on that came out rather ugly.

Here it is in plaintext:

---

$ pip install coreapi

[...]

Successfully installed coreapi-1.11.4

$ coreapi get https://api-sandbox.foxycart.com

<Error>

    errors: [

        {

            logref: "id-1453305072"

            message: "The FOXY-API-VERSION request header is required in order to use the API. Please include the following header with all requests: FOXY-API-VERSION: 1"

        }

    ]

    total: 1

$ coreapi headers add FOXY-API-VERSION 1  # Add a custom header to all outgoing requests.

Added header

Foxy-Api-Version: 1

$ coreapi get https://api-sandbox.foxycart.com

<Your API starting point. "https://api-sandbox.foxycart.com/">

    message: "Welcome to the FoxyCart API! Our hope is to be as self-documenting and RESTful as possible. Please let us know if you have any questions by posting in our forum http://forum.foxycart.com or emailing us at help...@foxycart.com. As a last resort, you could read the documentation at http://wiki.foxycart.com. Your first action should be to create an OAuth Client, then a user, followed by a store."

    create_client()  # Note that HAL APIs won't include the parameter listings here.