Siren, a Hypermedia Specification - Request for Review and Feedback

[Kevin Swiber]

Howdy, folks!

I recently started piecing together a specification of a hypermedia type.  I'd love to get your thoughts and ideas for improvement.  Many of your conversations here and your related work have been an inspiration.

First draft: https://github.com/kevinswiber/siren

I've been working on a Hypermedia API that has gone through several iterations.  Siren is a result of my latest efforts.

Some of the goals have been to represent, in a meaningful way:

- Communication of current state

- Embedded resources

- Embedded resource links

- Navigation links

- Actions

I developed Siren with an Entity concept, defined as being URI addressable and containing sub-entities, actions, and navigational links.

I have been focusing on JSON Siren as the first implementation, because it fits in nicely with my technology stack (Node.js, CouchDB, Web frontend).

Head on over to https://github.com/kevinswiber/siren to take a peek!  I'm happy to have discussions, and I'm also open to GitHub Issues and Pull Requests.

Thanks!

--
Kevin Swiber

Projects: https://github.com/kevinswiber
Twitter: @kevinswiber

[mca]

very interesting stuff; thanks for posting.

do you have any clients build for this design yet? an explorer, maybe?

is the format stable at this point or do you forsee significant changes soon?

mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Kevin Swiber]

I need to do some tweaking to an existing client to get it ready.  I wanted to release the ideas sooner than later to start collecting responses.  I'm hoping to get to the tooling this weekend.  The client/server reference implementation will likely be written in JavaScript (cross platform, quick to implement in Node, etc.).  

As far as stability, it has gone through many iterations in my brain, but I would still consider the format experimental.  Stability, in my opinion, comes from battle-testing via real-world usage and community involvement.  I'm still converting my Hypermedia API over to Siren, and the service for which I'm doing this is not yet public.

Thanks for taking a look.  I'm very open to suggestions for improvement or challenges of the current design.

[mca]

Kevin:

Looks good from here. seeing clients, explorers, etc. helps me get a handle the the foibles of the design.

two tings:
1 - i definitely encourage you to register this type w the IANA. not very difficult and very helpful.
2 - i encourage you to post this to the hypermedia-web list as there are also some very smart folks tehre who might contribute additional feedback.

looking forward to seeing more about this soon.

 
mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Kevin Swiber]

Hey Mike,

I appreciate the info.  I'll look into the IANA registration.

Also, not sure how I missed the existence of hypermedia-web.  I just joined and look forward to the discussions.

Thanks!

[Glenn Block]

Hi Kevin

Cool to see this. Have you looked at Mike Amundsen's Collection + JSON or Mike Kelly's HAL + JSON?

Interested to know how Siren is different....

Thanks

Glenn

[Kevin Swiber]

Hey Glenn,

Sure, I've seen both Collection+JSON and HAL.  Both are certainly good projects that are well-defined and have their place.  I'm not an expert in either format, so feel free to correct errors.

The first difference is conceptual.  Siren clearly defines resource representations as "entities," which possess certain characteristics.  Entities may have properties defining their state, sub-entities that are related, actions (or behaviors) that may result in state transitions, and links for navigating outside the entity graph.  Collection+JSON's goal is to provide fetch/update semantics for simple lists.  HAL defines resources that contain links, properties, and embedded resources.  HAL's main focus is linking.

Some areas where Siren differs:  

- Support for complex writes via actions that can define a method (PUT, PATCH, POST), encoding type, and URI.  

- Embedded links, in the context of a parent entity, that have the option of auto-dereferencing by the client.

- Classes for describing the nature of entities and actions in the context of the current representation (opposed to link relations that define relationships).

- A distinction between sub-entities and navigational links.

- Prevents property name collision by encapsulating all properties in a separate object ( { "properties": { ... } }).

There are likely more differences, but that's what I can think of off the top of my head.

These are all needs I had that I couldn't find in an existing format that also carries the same entity semantics and supports JSON.

Hope that helps.  Let me know if you have any additional questions.

Thanks!

[Mike Kelly]

Hey Kevin,

Couple of observations/questions in line

On Sat, Jul 7, 2012 at 5:24 AM, Kevin Swiber <ksw...@gmail.com> wrote:
> Hey Glenn,
>
> Sure, I've seen both Collection+JSON and HAL. Both are certainly good
> projects that are well-defined and have their place. I'm not an expert in
> either format, so feel free to correct errors.
>
> The first difference is conceptual. Siren clearly defines resource
> representations as "entities," which possess certain characteristics.
> Entities may have properties defining their state, sub-entities that are
> related, actions (or behaviors) that may result in state transitions, and
> links for navigating outside the entity graph.

fwiw, that's pretty much equivalent to HAL other than 'actions' which
were purposely left out.

> - Embedded links, in the context of a parent entity, that have the option of
> auto-dereferencing by the client.

Didn't quite grok this, could you explain it some more?

> - A distinction between sub-entities and navigational links.

is this the same as _embedded vs _links in HAL or something different?

Cheers,
M

[Kevin Swiber]

Hey Mike,

For the sake of comparison, I created a gist with the orders collection example from the HAL draft and added a potential JSON Siren equivalent.  https://gist.github.com/3066768

Where HAL adds customer and basket to the links array, Siren prefers to add these to the entities array as embedded links [1].

HAL's _embedded and _links vs. Siren's entities and links: Siren will represent links to entities inside the entities array, whereas HAL will put this reference in the _links array.  Siren's links array prefers rel values that are navigational in nature (up, next, previous, first, self, last, index, etc.).

One thing you'll notice right away looking at the gist is that HAL exercises a lot more brevity.  This is most likely a deliberate choice you made in the design.  Siren is a little bulkier, due to requirements such as:

- Links should be able to contain multiple rel values (while HAL doesn't explicitly prevent this, I believe the key-value approach with _links makes this more difficult to parse)

- Properties of the format should not collide with implementation-provided properties

- Entities and actions may have class attributes to describe the nature of the content in the context of the representation.

- Action structures are used even for operations such as search, whereas HAL utilizes URI templates.

I appreciate you taking a look, and I hope that answers your questions.

[1] http://www.amundsen.com/hypermedia/hfactor/#le

[Darrel Miller]

Kevin,

Where are the semantics of specific class values communicated?  In hal everything is left to link relations.  In ALPS stuff there is a "profile" document that can be pointed to in various ways.   How does this work in siren?

Darrel

[Mike Kelly]

Hey Kevin, thanks for clarifying - I have a couple more questions..

On Sat, Jul 7, 2012 at 4:37 PM, Kevin Swiber <ksw...@gmail.com> wrote:
> Hey Mike,
>
> For the sake of comparison, I created a gist with the orders collection
> example from the HAL draft and added a potential JSON Siren equivalent.
> https://gist.github.com/3066768
>
> Where HAL adds customer and basket to the links array, Siren prefers to add
> these to the entities array as embedded links [1].
>
> HAL's _embedded and _links vs. Siren's entities and links: Siren will
> represent links to entities inside the entities array, whereas HAL will put
> this reference in the _links array. Siren's links array prefers rel values
> that are navigational in nature (up, next, previous, first, self, last,
> index, etc.).

Ok, how does one determine whether a the link is considered
"navigational" or not?

What benefit is there in forcing this distinction?

> - Properties of the format should not collide with implementation-provided
> properties

I think that is a theoretical difference, properties are extremely
unlikely to collide with _links or _embedded in the case of HAL. The
underscore prefix is arguably even overkill.

> - Entities and actions may have class attributes to describe the nature of
> the content in the context of the representation.

Really not a fan of this approach (it borders on anti-pattern) but, if
you really have to, this is exactly what the "profile" link is
designed for - fwiw this is actually included in the hal+json spec.

> - Action structures are used even for operations such as search, whereas HAL
> utilizes URI templates.

How come?

Cheers,
M

[Kevin Swiber]

Good catch!  Honestly, I didn't document this, because I had some questions on the design that I never fully answered.

Prior to publishing the document on GitHub, I had a section that defines a meta object on the root entity.

{

  "meta": {

    "profile": "http://x.io/docs/slick-api"

  }

}

The questions is... Should any other piece of information exist in the meta object?  If the answer is no, then this is better represented as a profile property under the root entity.  If the answer is yes, what are those properties?

Any thoughts?

[Kevin Swiber]

Mike,

Great questions.  Responses below.

On Sat, Jul 7, 2012 at 12:27 PM, Mike Kelly <mikeke...@gmail.com> wrote:

Hey Kevin, thanks for clarifying - I have a couple more questions..

On Sat, Jul 7, 2012 at 4:37 PM, Kevin Swiber <ksw...@gmail.com> wrote:
> Hey Mike,
>
> For the sake of comparison, I created a gist with the orders collection
> example from the HAL draft and added a potential JSON Siren equivalent.
> https://gist.github.com/3066768
>
> Where HAL adds customer and basket to the links array, Siren prefers to add
> these to the entities array as embedded links [1].
>
> HAL's _embedded and _links vs. Siren's entities and links: Siren will
> represent links to entities inside the entities array, whereas HAL will put
> this reference in the _links array.  Siren's links array prefers rel values
> that are navigational in nature (up, next, previous, first, self, last,
> index, etc.).

Ok, how does one determine whether a the link is considered
"navigational" or not?

What benefit is there in forcing this distinction?

It's the difference between communicating options for traversal through the entity graph (via parent-child relationships, for instance) and providing the client options to move through your API "sitemap."  Encapsulation is a heavily used concept in Siren. There is an implied ownership with sub-entities that doesn't exist in the links structure.  That ownership in HAL would have to be communicated via documentation about the link relation.  I believe forcing this distinction actually results in a reduction of complexity via a clearer understanding of the link's context (sub-entity vs. higher level navigation).  Do you think there's a disadvantage to this approach?

 

> - Properties of the format should not collide with implementation-provided
> properties

I think that is a theoretical difference, properties are extremely
unlikely to collide with _links or _embedded in the case of HAL. The
underscore prefix is arguably even overkill.

As long as it's a valid JSON property name, there is the potential for collision.  I'm more comfortable making this a non-issue over a "not likely to happen" approach.  Just a difference in trade-offs and design.  The trade-off for JSON Siren is more verbose syntax with the properties object.  I feel that's acceptable.

 

> - Entities and actions may have class attributes to describe the nature of
> the content in the context of the representation.

Really not a fan of this approach (it borders on anti-pattern) but, if
you really have to, this is exactly what the "profile" link is
designed for - fwiw this is actually included in the hal+json spec.

Can you elaborate?  I'd like to know why you see it as an anti-pattern.  I've struggled with this one, myself, so I'd love to get your opinion.

 

> - Action structures are used even for operations such as search, whereas HAL
> utilizes URI templates.

I need actions for write/delete operations, as well.  Also, the fields of actions define types that can be used for validation by the client.  Again, this information would have to be communicated outside of the document itself, which can be acceptable, but it doesn't allow the server to change the types on the fly without potentially breaking clients.  Do you think URI templates are a better way to accomplish this?

Thanks,

[Pat Cappelaere]

Would you consider using an API service document that would publish the service metadata, resources, schemas, security... ?
This might help a lot.
Some of the actions may not have to be duplicated for each entry.  For example, if you use OpenSearch, you have a discovery document that would specify how to find orders.
An API discovery document would be a similar extension akin to what Google is currently doing.
Very interesting topic.
Thank you.
Pat.

On Saturday, July 7, 2012 12:36:15 PM UTC-4, Kevin Swiber wrote:

Good catch!  Honestly, I didn't document this, because I had some questions on the design that I never fully answered.

Prior to publishing the document on GitHub, I had a section that defines a meta object on the root entity.

{

  "meta": {

    "profile": "http://x.io/docs/slick-api"

  }

}

The questions is... Should any other piece of information exist in the meta object?  If the answer is no, then this is better represented as a profile property under the root entity.  If the answer is yes, what are those properties?

Any thoughts?

[Mike Schinkel]

Hi Kevin, 

Nice effort on Siren.  Now we have ~7 ways to do this. :)

On Jul 7, 2012, at 1:05 PM, Kevin Swiber wrote:

On Sat, Jul 7, 2012 at 12:27 PM, Mike Kelly <mikeke...@gmail.com> wrote:

It's the difference between communicating options for traversal through the entity graph (via parent-child relationships, for instance) and providing the client options to move through your API "sitemap."  Encapsulation is a heavily used concept in Siren. There is an implied ownership with sub-entities that doesn't exist in the links structure.  That ownership in HAL would have to be communicated via documentation about the link relation.  I believe forcing this distinction actually results in a reduction of complexity via a clearer understanding of the link's context (sub-entity vs. higher level navigation).  Do you think there's a disadvantage to this approach?

 

> - Properties of the format should not collide with implementation-provided
> properties

I think that is a theoretical difference, properties are extremely
unlikely to collide with _links or _embedded in the case of HAL. The
underscore prefix is arguably even overkill.

As long as it's a valid JSON property name, there is the potential for collision.  I'm more comfortable making this a non-issue over a "not likely to happen" approach.  Just a difference in trade-offs and design.  The trade-off for JSON Siren is more verbose syntax with the properties object.  I feel that's acceptable.

Be aware that the approach HAL and some others take allows them to be applied to an existing API in a manner that would be backward compatible for clients and thus client could progressively support the new functionality, but with your approach clients of existing APIs would need to be at least somewhat rewritten to still work if the API owner decided to adopt Siren.  

So my feedback is: for use-cases I care about the explicit 'properties' makes selecting Siren a non starter.  Just saying.

-Mike

[Mike Schinkel]

On Jul 7, 2012, at 4:56 PM, Pat Cappelaere wrote:
> Would you consider using an API service document that would publish the service metadata, resources, schemas, security... ?
> This might help a lot.
> Some of the actions may not have to be duplicated for each entry. For example, if you use OpenSearch, you have a discovery document that would specify how to find orders.
> An API discovery document would be a similar extension akin to what Google is currently doing.
> Very interesting topic.

Agreed. A API discovery document would make it a lot more appealing to me too. Especially if one why to find it would be to look in robots.txt for the link and thus be able to leave existing APIs returning JSON unchanged.

-Mike

[Pat Cappelaere]

You should find it in the head of the home page as a link (same as open search)
Right?
Pat

Sent from my iPad

[mca]

pat:

that's pretty much the "profile" pattern[1]; quite handy, IMO.

[1] http://gmpg.org/xmdp/

mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Mike Schinkel]

On Jul 7, 2012, at 7:45 PM, mca wrote:

that's pretty much the "profile" pattern[1]; quite handy, IMO.

[1] http://gmpg.org/xmdp/

Just curious, how would you envision using "profile" with JSON?  Do you mean using "profile" in the HTTP "Content-type" header like [1], colloquially to be like the pattern used by Xhtml Meta Data Profiles, or something else?

-Mike

[1] http://tools.ietf.org/html/draft-zyp-json-schema-02#section-4

[Kevin Swiber]

Thanks, Mike.  You're absolutely right.  Siren, as it is now, does not support building on an existing API and maintaining backwards compatibility.  Do you feel strongly that it should be a goal of the project?

To be perfectly honest, it's a use case I didn't even consider.  Shifting to a completely new media type is a very large change that I would expect to break an existing API.  If such a change was warranted, I would take a different approach, preferring to support multiple media types while having human conversations to discuss a transition strategy for existing clients (assuming these clients' developers are human-reachable, of course. :)

Thanks for the feedback!

[Steve Klabnik]

> Just curious, how would you envision using "profile" with JSON? Do you mean
> using "profile" in the HTTP "Content-type" header like [1], colloquially to
> be like the pattern used by Xhtml Meta Data Profiles, or something else?

http://tools.ietf.org/html/draft-wilde-profile-link-01

[Pat Cappelaere]

May be handy if you are a web page and need to add some metadata about it.

I am not quite sure that I would use this to describe a web service API to be mined by an external App.

Pat.

On Jul 7, 2012, at 7:45 PM, mca wrote:

[Mike Schinkel]

On Jul 7, 2012, at 9:03 PM, Pat Cappelaere wrote:
> May be handy if you are a web page and need to add some metadata about it.
> I am not quite sure that I would use this to describe a web service API to be mined by an external App.

What's your opinion, and others, about having an API discovery doc at a /.well-known/ location[1]?

-Mike
[1] http://tools.ietf.org/html/rfc5785

[Pat Cappelaere]

I think it is critical to have all that info in one place and auto-discoverable like this:
http://www.opensearch.org/Specifications/OpenSearch/1.1/Draft_3#Autodiscovery
Note: that this not a well known location but discoverable from a known location :)

I really like the Google API Discovery Service Document: https://developers.google.com/discovery/v1/reference

Pat.

[Pat Cappelaere]

Steve,
I understand that one could but why would you add all that overhead in every resource representation?
Pat.

[Steve Klabnik]

One single Link header is not a ton of overhead.

[Pat Cappelaere]

times the number of resources?

[Steve Klabnik]

It's per request.

[Patrice Cappelaere]

And you would not use that for a list of resources?

-- 
Patrice Cappelaere

Sent with Sparrow

On Saturday, July 7, 2012 at 9:15 PM, Steve Klabnik wrote:

It's per request.

[Steve Klabnik]

I do not think that one single extra link per request is a significant
amount of 'overhead.' I'd much rather do it that way than encourage
some kind of out-of-band 'well known' location just to save a few
bytes.

I'm sure that all the </a> tags and </h1> tags in a web request add
considerably more overhead than a single link. Granted, gzip may come
into play here, but still, it seems like an absolutely trivial
concern.

I actually also really despise 'service documents,' so it may just be
that we come from totally opposite worlds in terms of taste. The web
is supposed to be loosely coupled, 'dynamically typed,' if you will.
Service documents and well-known locations feel like the opposite of
what the web is supposed to be about.

[Mike Schinkel]

Hi Kevin,

On Jul 7, 2012, at 8:18 PM, Kevin Swiber wrote:

Thanks, Mike.  You're absolutely right.  Siren, as it is now, does not support building on an existing API and maintaining backwards compatibility.  Do you feel strongly that it should be a goal of the project?

To be perfectly honest, it's a use case I didn't even consider.  Shifting to a completely new media type is a very large change that I would expect to break an existing API.  If such a change was warranted, I would take a different approach, preferring to support multiple media types while having human conversations to discuss a transition strategy for existing clients (assuming these clients' developers are human-reachable, of course. :)

Siren might be absolutely perfect for your needs so I can't speak to your use-cases, I can only speak to the ones that interest me; Mark Nottingham's "MANY CLIENTS AND ONE SERVER" [1]. But since you requested feedback I assume you were asking for how it applies to our use-cases?  If yes, then...

What I see is that public APIs are almost always designed for ease of consumption. IMO this is because the more complexity that an API client must process, the less adoption it is likely to see.  So most public web APIs publishers support URL construction and really simple JSON responses. I think getting API publishers to support more elaborate media types is an uphill battle.  I think people start building APIs by building something that is as simple as possible, and then they grow their APIs in complexity as the number and different types of clients using the API grows. 

This IMO the best approach is one where the complexity can be optionally layered and grown over time, especially if the complexity can be stored in a discovery document so the existing API isn't changed and only the clients that need more complexity have to interact with it.

Lastly, I lament the fact there are so many different competing approaches each with their own one or two advocates; to name a few: HAL, JSON-LD, OData, Object Network, Collection+JSON, JSON Schema plus several others I can't think of at the moment, and now Siren.  What I'd really like to see is everyone who's advocating their own approach come together and agree to collaborate on a NEW approach where all the advocates agree up front that the goal is to create a signal approach that can become a standard, and ideally one that can start very simple but evolve to handle more and more complexity as more and more of the web adopts it.  

If that one standard approach can emerge, then I believe we're likely to see open-source client and server libraries in major web languages/platforms, explorers in browser's developer toolkits, and maybe the equivalent of XMLHttpRequest but for this new standard to be added to all browsers, sooner than later.

FWIW. :)

-Mike

[1] http://www.mnot.net/blog/2012/06/25/http_api_complexity_model

[Mike Schinkel]

On Jul 7, 2012, at 8:25 PM, Steve Klabnik wrote:
>> Just curious, how would you envision using "profile" with JSON? Do you mean
>> using "profile" in the HTTP "Content-type" header like [1], colloquially to
>> be like the pattern used by Xhtml Meta Data Profiles, or something else?
>
> http://tools.ietf.org/html/draft-wilde-profile-link-01

Thanks. But as I understand it that spec refers to a <link> relation, which is X/HTML. Do I understand correctly that the API domain must serve an X/HTML document to offer this? What about domains where the APi developer doesn't have access to update the HTML? What about services that would rather serve JSON alone?

Maybe I misunderstand?

-Mike

[Steve Klabnik]

> Thanks. But as I understand it that spec refers to a <link> relation, which is X/HTML. Do I understand correctly that the API domain must serve an X/HTML document to offer this? What about domains where the APi developer doesn't have access to update the HTML? What about services that would rather serve JSON alone?

Link relations don't have to just be for X/HTML, you can also use the
Link header to add it to any kind of response.

http://tools.ietf.org/html/rfc5988#section-5

[Mike Schinkel]

But as described in the RFC it can only be in the HTTP response body if X/HTML, right?

On Jul 7, 2012, at 9:36 PM, Steve Klabnik wrote:
> I actually also really despise 'service documents,' so it may just be
> that we come from totally opposite worlds in terms of taste. The web
> is supposed to be loosely coupled, 'dynamically typed,' if you will.
> Service documents and well-known locations feel like the opposite of
> what the web is supposed to be about.

Something to consider; service documents can potentially describe existing (somewhat) RESTful web APIs without requiring them to be modified. There is a potential benefit it allowing existing APIs to be described via a single service document at a well known location because that lowers the bar for them to become more RESTful and more traversable.

Since APIs by best-practice should be slow-to-change I'm not sure having a document that provides information about an existing API is really that diametrically opposed to what the web is about. I will definitely give you that most mentions of discovery documents refer to heavily architected efforts and thus the term bears negative connotations to many, but that's not what I was discussing.

Instead I'm envisioning discovery documents that start very simple with only the most basic components included and over time allowed to grow with community input just like HTML has grown over the years (albeit, hopefully faster than HTML has grown.)

-Mike

[Steve Klabnik]

> But as described in the RFC it can only be in the HTTP response body if X/HTML, right?

Any media type that uses link relations, at least as I've read it. If
that's not the case, well, then another bug should get filed before
that ID turns into an RFC. :)

> Instead I'm envisioning discovery documents that start very simple with only the most basic components included and over time allowed to grow with community input just like HTML has grown over the years (albeit, hopefully faster than HTML has grown.)

I mean this on a deeper, philosophical level than any particular
implementations. Service documents are fine for an RPC style API, but
not for a REST-y / Hypermedia one. Then again, I guess this list is
'API-craft,' not 'hypermedia-api-craft," eh? Maybe that's where this
divergence is coming from, actually. I'm on too many mailing lists. ;)

[Steve Klabnik]

Ah! I should point to this, too: http://bitworking.org/news/193/Do-we-need-WADL

This is basically what I'm in agreement with in regards to 'service
documents.' Is WADL the kind of thing you're talking about? Can you
point me to an existing document of this type?

[Mike Schinkel]

On Jul 7, 2012, at 10:12 PM, Steve Klabnik wrote:
> I mean this on a deeper, philosophical level than any particular
> implementations. Service documents are fine for an RPC style API, but
> not for a REST-y / Hypermedia one.

Respectfully, I disagree.

I think that service documents would be perfect for a RESTful API (although we *may* be talking about different things.) A RESTful service document could contain the same information that would be served up in a self-contained hypermedia RESTful response, but with the hypermedia information abstracted away into the service document using URL templates such that the actual responses contain only data. Another way to say it is the service document is essentially the RESTful API's documentation but in a form that can be processed in an automated manner.

I don't see any reason why hypermedia RESTful APIs are any different in *this* respect than "RPC-style" ones, except the former's service documents define nouns and the latter's verbs.

-Mike

[Steve Klabnik]

> I think that service documents would be perfect for a RESTful API (although we *may* be talking about different things.) A RESTful service document could contain the same information that would be served up in a self-contained hypermedia RESTful response, but with the hypermedia information abstracted away into the service document using URL templates such that the actual responses contain only data. Another way to say it is the service document is essentially the RESTful API's documentation but in a form that can be processed in an automated manner.

This may be the difference, yes. :) When I think 'service document' I
think 'documents the entire service,' not just 'the stuff that the
current representation links to.' Would that make sense, then? Is this
how I'm misunderstanding you?

Sorry for being a bit dense. :/

[mca]

On Sat, Jul 7, 2012 at 10:27 PM, Mike Schinkel <mi...@newclarity.net> wrote:

<snip>

I think that service documents would be perfect for a RESTful API (although we *may* be talking about different things.)  A RESTful service document could contain the same information that would be served up in a self-contained hypermedia RESTful response, but with the hypermedia information abstracted away into the service document using URL templates such that the actual responses contain only data. 

</snip>

I'd love to see a mock of this, see how it would work in an existing API and how it might affect coding clients (if any).

got an example i can check out?

mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Mike Schinkel]

Yes. The problem is I'm probably misusing the term in the way most people use it.  What I really mean is to extract the hypermedia control information into a single resource for the API instead of including it in every JSON API response.

Sorry for being a bit dense. :/

LOL! No worries, I have my fair share of being far more dense than that. 

On Jul 7, 2012, at 10:40 PM, mca wrote:

I'd love to see a mock of this, see how it would work in an existing API and how it might affect coding clients (if any).

got an example i can check out?

You *would* call me on that, now wouldn't you. :)  

I started to mock a bit of the Twitter API earlier and realized I didn't know URI Template syntax well enough to get it perfect, but if you are willing to overlook my URI Template errors I could mock up a few resources.

Also, as I understand it oData JSON Light[1] can be used this way too, although I've not verified that.

-Mike

[1] http://bit.ly/odata-json-light

[mca]

<snip>

You *would* call me on that, now wouldn't you. :)  

I started to mock a bit of the Twitter API earlier and realized I didn't know URI Template syntax well enough to get it perfect, but if you are willing to overlook my URI Template errors I could mock up a few resources.

</snip>

LOL! yeah, i tend to do that, eh?

first, i think this is a very intriguing idea; i've seen it mentioned, but not seen a working model at runtime, only at design-time.

second, for early designs, i wouldn't sweat the details of URI Templates. getting a general layout, a clear way to associate the runtime data representation w/ the proper bits of hypermedia description info, and getting a sense of how it will look to a client dev is more interesting right now.

I know the ODATA guys have been working on service descriptions but what i've seen does not sound quite like what you describe; i encourage you to follow this up.

mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Pat Cappelaere]

Steve,

You may despise service documents but you do need to provide a lot of out-of-band information to a programmer so he can interface to your web service regardless. This does not affect any coupling, right?
What is the difference?
Wouldn't you want to provide the same information in a way that can be used directly by the client?
This would really help everyone and reduce many interpretation errors.
Pat.

On Jul 7, 2012, at 9:36 PM, Steve Klabnik wrote:

[Steve Klabnik]

> You may despise service documents but you do need to provide a lot of out-of-band information to a programmer so he can interface to your web service regardless.

You should ideally not provide any out-of-band information. That's the
whole idea behind the stateless constraint. :)

> What is the difference?

Between what, a service document and a media type?

[Pat Cappelaere]

Steve,

I am completely against the RPC style API and for a Hypermedia style API.  I really do not see a conflict with using a service document.  You need access to that out-of-band context one way or another.

Pat.

On Jul 7, 2012, at 10:12 PM, Steve Klabnik wrote:

[Pat Cappelaere]

LOL.

So you never provide any documentation regarding your web service?
Programmers can just start interfacing with it with no prior knowledge?
They can guess all the resources that are published, the security protocol, the various action methods, the semantic behind the links…?
There is no difference between documentation and a service document. There are both documentation about the service.

Pat.

[Pat Cappelaere]

You can use something like this: https://github.com/mashery/iodocs driven by a Google Discovery Document published by the web service.

You get an API tester for free.

Pat.

[Steve Klabnik]

> So you never provide any documentation regarding your web service?

I never said that.

> Programmers can just start interfacing with it with no prior knowledge?

Assuming they know how to interact with the media type, yes.

> They can guess all the resources that are published, the security protocol, the various action methods, the semantic behind the links…?

Assuming they know how to interact with the media type, yes.

> There is no difference between documentation and a service document. There are both documentation about the service.

I think we have a big issue in terminology here. It's a big
disconnect. Here's what I'm thinking:

"Service Document": some sort of document that describes an entire
document in depth, machine readable. WSDL and WADL are examples of
"service documents."
"out-of-band": information that is needed to handle a response that is
provided outside of the information given in the request.

Does this line up with what you're thinking, or are we talking around
two different things?

[Mike Schinkel]

On Jul 7, 2012, at 11:27 PM, mca wrote:

<snip>

I started to mock a bit of the Twitter API earlier and realized I didn't know URI Template syntax well enough to get it perfect, but if you are willing to overlook my URI Template errors I could mock up a few resources.

</snip>

first, i think this is a very intriguing idea; i've seen it mentioned, but not seen a working model at runtime, only at design-time.

second, for early designs, i wouldn't sweat the details of URI Templates. getting a general layout, a clear way to associate the runtime data representation w/ the proper bits of hypermedia description info, and getting a sense of how it will look to a client dev is more interesting right now.

So I created a really simple document that could exist at a hypothetical "https://api.twitter.com/.well-known/json-api.json" to represent two of the Twitter API resources:

https://gist.github.com/3069508

Since I spent 2 hours on this I didn't agonize over the architecture, I just built something to show.  

Then I started writing an explorer for it in PHP, and remember how difficult it is to write an API explorer in an evening. :)  Still, I'm pretty sure the data captured in the json-api.json would be enough to allow the response from the Twitter Search API to be surfed to the status API assuming someone had the time to write the explorer.  Hopefully you (mca) or someone else find this idea has value and can push it forward[*].

-Mike

[*]  I would love to move it forward but it's a too big of a project to just done on the side and my current clients are unfortunately not the kind interested in paying for this type of work.

[Mike Schinkel]

On Jul 7, 2012, at 11:52 PM, Steve Klabnik wrote:

> On Jul 7, 2012, at 11:48 PM, Pat Cappelaere wrote:
>> Programmers can just start interfacing with it with no prior knowledge?
>
> Assuming they know how to interact with the media type, yes.
>
>> They can guess all the resources that are published, the security protocol, the various action methods, the semantic behind the links…?
>
> Assuming they know how to interact with the media type, yes.
>
>> There is no difference between documentation and a service document. There are both documentation about the service.
>
> I think we have a big issue in terminology here. It's a big
> disconnect. Here's what I'm thinking:
>

> ...

>
> "out-of-band": information that is needed to handle a response that is
> provided outside of the information given in the request.

Then isn't knowledge of the media type "out-of-band" information?

-Mike

[Mike Kelly]

On Sun, Jul 8, 2012 at 4:48 AM, Pat Cappelaere <cappe...@gmail.com> wrote:
> LOL.
>
> So you never provide any documentation regarding your web service?
> Programmers can just start interfacing with it with no prior knowledge?
> They can guess all the resources that are published, the security protocol, the various action methods, the semantic behind the links…?
> There is no difference between documentation and a service document. There are both documentation about the service.

Hi Pat,

Try this:

http://haltalk.herokuapp.com/

Cheers,
Mike

[Jon Moore]

On Jul 8, 2012, at 1:24 AM, Mike Schinkel <mi...@newclarity.net> wrote:

Then I started writing an explorer for it in PHP, and remember how difficult it is to write an API explorer in an evening. :)

As an aside, one of my favorite reasons for using HTML as the media type for an API is never having to write an API explorer. :)

Jon

[Mike Kelly]

Ok, but a browser for a media type is not something you necessarily
need to (re)implement yourself, it should be re-usable between
applications.

HTML is not a good option for m2m APIs, it's bloated with features
that have no relevance to a machine interface.. using it increases the
challenge of interacting with your app, making it more likely that
people will go elsewhere.

Cheers,
M

[Jon Moore]

On Sun, Jul 8, 2012 at 5:27 AM, Mike Kelly <mikeke...@gmail.com> wrote:

Ok, but a browser for a media type is not something you necessarily
need to (re)implement yourself, it should be re-usable between
applications.

True, but why even have to implement it at all?

 

HTML is not a good option for m2m APIs, it's bloated with features
that have no relevance to a machine interface.. using it increases the
challenge of interacting with your app, making it more likely that
people will go elsewhere.

You can always restrict yourself (quite successfully) to HTML features that *are* relevant to m2m interactions, and end up with quite concise representations. We are actively doing this, by the way, and it is working well.

Can you talk a little more about your opinion of the relative challenges of:

* Bootstrapping a new media type (e.g. Siren or HAL) vs. using a media type with a huge population of people who understand it (HTML)

* Getting new developers to download and install a media type explorer vs. having one already installed (web browser)

I would assert that the real challenge here is actually getting people to build hypermedia-aware clients. If you don't want to build a hypermedia API, then I agree that HTML doesn't offer you much at all (but then neither do Siren or HAL when compared to plain application/json). If you *do* want to build hypermedia APIs, then I think overlooking a domain-agnostic hypermedia-aware media type with tons of existing tooling and a huge developer population and trying to mint your own is actually the bigger uphill climb. 

I'll grant that HTML isn't perfect (form support for something other than GET/POST, for example) and there aren't widespread client libraries for treating it as an API...yet. Personally, I prefer to start from HTML and build the (minimal) extra tooling I need while working through the standards bodies to fix its deficiencies over time. There is also the potential for adding the ENTIRE WEB as addressable/linkable surface area from your API, because your clients will already have the basic processing in place for it.

Now, please don't construe this as me saying that Siren or HAL aren't valuable. On the contrary, I think we are pretty far away from understanding what good, easily-adoptable hypermedia APIs look like, so I think there's value in exploring multiple options, because, after all, I might be wrong about using HTML! Really this is more of an attempt to help refine the arguments for using and creating media types like Siren or HAL.

Jon

--
........
Jon Moore

[Steven WIllmott]

On Jul 8, 2012, at 12:45 PM, Jon Moore wrote:

On Sun, Jul 8, 2012 at 5:27 AM, Mike Kelly <mikeke...@gmail.com> wrote:

Ok, but a browser for a media type is not something you necessarily
need to (re)implement yourself, it should be re-usable between
applications.

True, but why even have to implement it at all?

 

HTML is not a good option for m2m APIs, it's bloated with features
that have no relevance to a machine interface.. using it increases the
challenge of interacting with your app, making it more likely that
people will go elsewhere.

You can always restrict yourself (quite successfully) to HTML features that *are* relevant to m2m interactions, and end up with quite concise representations. We are actively doing this, by the way, and it is working well.

I think the problem isn't "why not HTML" it's "why HTML" - if you strip out all the parts of HTML which are to do with rendering things for presentation you're left with almost nothing at all:

<a>

<h1>, <h2>, <h3> ... (as nesting)

<ul><li>

<ol><li>

<p> maybe (as a kind of separator) - or <div>

...

and even some of these are marginal. There is useful stuff around encodings, meta-data etc. but pretty much everything else is redundant. Hence all that benefit of having browsers already implemented is almost entirely lost because the majority of what's in a browser implementation is there to deal with the stuff you don't care about. 

If you wanted to go this way then at least you'd want to use XML so you had a way to attach object/data schemas to the content of the interactions.

Can you talk a little more about your opinion of the relative challenges of:

* Bootstrapping a new media type (e.g. Siren or HAL) vs. using a media type with a huge population of people who understand it (HTML)

* Getting new developers to download and install a media type explorer vs. having one already installed (web browser)

I would assert that the real challenge here is actually getting people to build hypermedia-aware clients. If you don't want to build a hypermedia API, then I agree that HTML doesn't offer you much at all (but then neither do Siren or HAL when compared to plain application/json). If you *do* want to build hypermedia APIs, then I think overlooking a domain-agnostic hypermedia-aware media type with tons of existing tooling and a huge developer population and trying to mint your own is actually the bigger uphill climb. 

I'll grant that HTML isn't perfect (form support for something other than GET/POST, for example) and there aren't widespread client libraries for treating it as an API...yet. Personally, I prefer to start from HTML and build the (minimal) extra tooling I need while working through the standards bodies to fix its deficiencies over time. There is also the potential for adding the ENTIRE WEB as addressable/linkable surface area from your API, because your clients will already have the basic processing in place for it.

The entire web does indeed use HTML, but 98% of the traffic that's passing back and forth across those channels is rendering / presentation information which you want to ditch/abstract away. Sure it's nice to build an engine to interact with an existing website - people do this all the time to screen scrape (and companies like Kapow and Dapper built very sophisticated robots to do this in automated ways) - but most of the hard work is do with stripping out the cruft you don't need to perform the operations you do.

Lastly the most important argument for me is that if you set out to use the browsers implementation out there you might end up stuck with having to support them (or a subset of them) to keep working for a use case which they are not really designed for - which is just a nightmare scenario.

Now, please don't construe this as me saying that Siren or HAL aren't valuable. On the contrary, I think we are pretty far away from understanding what good, easily-adoptable hypermedia APIs look like, so I think there's value in exploring multiple options, because, after all, I might be wrong about using HTML! Really this is more of an attempt to help refine the arguments for using and creating media types like Siren or HAL.

I wouldn't say it's invalid but rather like a "lets make horses faster/better" type of approach, sometimes starting over has a lot of benefits. Especially when it means a lot of simplification.

 steve.

[Pat Cappelaere]

Steve,

We are talking about the same thing. A "Service Document" does provide enough contextual information to get a developer going and start interfacing with your service.
I am certainly not a big fan of WSDL (and the WS-* stack) nor WADL for that matter…
But you need to define your media-types somehow.
Mike Kelly rightfully complained that we have now seven different ways of representing a resource in JSON. I certainly would love to see everyone coming to the table and agreeing… but we get into religion wars…and personal preferences… The multiplicity of media types does not help here: application/json+siren, application/hal+json, application/vnd.collection+json or application/my-app+json. These are all application/json. It just confuses the existing tools without adding any consequent information.
The pragmatic way is to let people do it the way they want as long as they describe their resource representations properly. If they use JSON, then use JSON schema. If they use XML, then use XML schema
View this as documentation you need to do anyway. You can use it to generate your user doc and validate user entries. At least, it will stay in sync with your evolving API (not a guarantee if you have separate docs). This is what a service document does.

I have a lot of respect for Jon Moore and HTML but… any decent HTML page these days will have a lot of javascript, jquery callbacks, social media tracking and so on… That's fine to render a view targeted to a human. But, if you output a representation for a machine, it may behoove you to use something more appropriate. This could be a bare bone HTML but then why not use XML or JSON? HTML scraping is not a best practice. Simpler may be better.

Would we be in agreement and say:
- Web users are not just humans but more and more machines.
- Hypermedia API is here to stay. No question about it. Machines need to be able to follow links AND understand what they are following. That's the real challenge.
- m2m rendering is a must. Use what is appropriate for your needs but document it in a way that can be used m2m as well. I do not see a way around some kind of a service document and simple schema. We need to use it for our docs and internal validation. It has to stay current as the app evolves over time.
- Custom media-types seem to be very high risk. Is this really worth it?

These are all great topics of discussion :)

Pat.

[Kevin Swiber]

On Sat, Jul 7, 2012 at 9:38 PM, Mike Schinkel <mi...@newclarity.net> wrote:

Siren might be absolutely perfect for your needs so I can't speak to your use-cases, I can only speak to the ones that interest me; Mark Nottingham's "MANY CLIENTS AND ONE SERVER" [1]. But since you requested feedback I assume you were asking for how it applies to our use-cases?  If yes, then...

Many Clients, One Server is my use case, as well.

 

What I see is that public APIs are almost always designed for ease of consumption. IMO this is because the more complexity that an API client must process, the less adoption it is likely to see.  So most public web APIs publishers support URL construction and really simple JSON responses. I think getting API publishers to support more elaborate media types is an uphill battle.  I think people start building APIs by building something that is as simple as possible, and then they grow their APIs in complexity as the number and different types of clients using the API grows. 

It's only an uphill battle, because the buy-in of both understanding and using Hypermedia API's has not crossed the chasm.

 

This IMO the best approach is one where the complexity can be optionally layered and grown over time, especially if the complexity can be stored in a discovery document so the existing API isn't changed and only the clients that need more complexity have to interact with it.

Hypermedia types should allow API complexity to grow over time. The media type has to support API evolution, and so building a client for a hypermedia type can be complex.  It's a trade-off.  A complex media type client allows your API to evolve.

There's a distinction here between a "media type client" and an "API client."  In the real world, that "media type client" is likely a library an API client developer can use to communicate with the API.  Once the media type is stable and standardized, this library will rarely, if ever, change.  It's then up to the API designer to adhere to those media type principles that help support evolution.  

The API client is responsible for implementing the bits you might find in a profile, such as rel or class definitions.  In practice, the API client can be relatively simple, assuming your hypermedia type library is well-built and very helpful.  The API client should only have to opt-in to additional complexity that comes from API evolution.  With hypermedia principles, old versions of clients will still work even as the API grows.  That's the end goal.

Sorry for being long-winded here, but I think there are a lot of misconceptions.  

 

Lastly, I lament the fact there are so many different competing approaches each with their own one or two advocates; to name a few: HAL, JSON-LD, OData, Object Network, Collection+JSON, JSON Schema plus several others I can't think of at the moment, and now Siren.  What I'd really like to see is everyone who's advocating their own approach come together and agree to collaborate on a NEW approach where all the advocates agree up front that the goal is to create a signal approach that can become a standard, and ideally one that can start very simple but evolve to handle more and more complexity as more and more of the web adopts it.  

If that one standard approach can emerge, then I believe we're likely to see open-source client and server libraries in major web languages/platforms, explorers in browser's developer toolkits, and maybe the equivalent of XMLHttpRequest but for this new standard to be added to all browsers, sooner than later.

I don't think implementing the Highlander principle is as valuable as it seems here.  Hypermedia is just starting to be understood in concept on a wider scale, and that's why there are several efforts.  While there may be some overlap, each of these types has a different set of goals.  The state of hypermedia understanding and adoption makes it perfect to have several ideas swimming in the wild. Collaboration and fall-out will come naturally over time.  I don't think only one will emerge as a "victor."  I'm excited to see what happens over the next few months and years in this arena.

FWIW. :)

Much appreciated.   :)

--
Kevin Swiber

Projects: https://github.com/kevinswiber
Twitter: @kevinswiber

[Kevin Swiber]

On Sat, Jul 7, 2012 at 8:25 PM, Steve Klabnik <st...@steveklabnik.com> wrote:

> Just curious, how would you envision using "profile" with JSON?  Do you mean
> using "profile" in the HTTP "Content-type" header like [1], colloquially to
> be like the pattern used by Xhtml Meta Data Profiles, or something else?

http://tools.ietf.org/html/draft-wilde-profile-link-01

Honestly, I had no idea this Internet-Draft existed.  Love the idea of profile as a link relation.  Makes total sense.  Thanks for sharing.

[Mike Kelly]

On Sun, Jul 8, 2012 at 4:43 AM, Steve Klabnik <st...@steveklabnik.com> wrote:
>> You may despise service documents but you do need to provide a lot of out-of-band information to a programmer so he can interface to your web service regardless.
>
> You should ideally not provide any out-of-band information. That's the
> whole idea behind the stateless constraint. :)

No, the stateless constraint is specifically about shared state
between client and server.

If you write an automated client that performs POST requests via <link
rel="widgets-collection" href="..." /> the semantics are out-of-band
in the documentation for the rel widgets-collection, but the stateless
constraint is not broken. There's also nothing wrong with that
approach, it's a design decision, and it happens to be a reasonable
one for machine-consumed applications or 'APIs'.

Cheers,
M

[Steve Klabnik]

> No, the stateless constraint is specifically about shared state
> between client and server.

I shouldn't have said 'entire idea.' That's what I get for posting
late at night. ;)

I'm specifically referring to "such that each request from client to
server must contain all of the information necessary to understand the
request," when I say this. Of course, session state is also a huge
part of that constraint.

And it'd actually probably be better to file it under the
'self-descriptive messages' sub-constraint of the uniform interface
constraint, "REST enables intermediate processing by constraining
messages to be self-descriptive: interaction is stateless between
requests, standard methods and media types are used to indicate
semantics and exchange information, and responses explicitly indicate
cacheability." This sentence also demonstrates a relationship between
statelessness and self-descriptive messages.

Point is, out-of-band information is bad. :)

[Steve Klabnik]

> Thanks for sharing.

Glad to point it out. Hope it makes it through the process to become an RFC.

[Steve Klabnik]

> Then isn't knowledge of the media type "out-of-band" information?

This is why terminology is hard. :/

Obviously, the processing rules are not included with the message
itself. However, by pointing at a media type, we can say "If you
understand text/html, you know _everything_ you need to know." Extra
semantics outside of text/html would be out-of-band. Media type
documentation is obviously not machine-readable, and a human will need
to read them. But you can treat the Content-Type string as a pointer.
A simple substitution.

The practical point of this is to allow clients and servers to evolve
independently of one another. Out-of-band info requires the two
parties to share information outside the protocol itself, which means
they need to be in touch with each other. If everything is in the
media type, clients and servers can be implemented without knowing
about each other's details.

[Mike Kelly]

Again that isn't what the self-descriptive constraint is about,
either. The quote you picked uses the term 'indicating' semantics
rather than 'establishing', and it doesn't make any mention of
out-of-band or otherwise.

Not even the "hateoas" constraint includes constraint about about
out-of-band information.

> Point is, out-of-band information is bad. :)

That sounds good, but it's not actually true - it completely depends
on the application and your design objectives.

Cheers,
M

[Mike Schinkel]

Four things:

1.) HTML is notoriously difficult to parse whereas JSON is absolutely trivial to parse in most web languages/platforms.  XHTML is easier, but still no where near as easy as JSON is some web languages/platforms.

2.) X/HTML response representations are bloated compared to their JSON equivalent and that's an issue for mobile clients.

3.) HTML didn't really apply here as you referred to my post about a proof-of-concept "service document" that defined rules for processing Twitter's JSON based API, and 

4.) Ironically the difficult part of writing the service document was not the display to the use but instead presenting the actions that could be performed by the API. I would have had the exact same difficulty if I had encoded the service document in HTML.

-Mike

[Mike Schinkel]

On Jul 8, 2012, at 10:55 AM, Steve Klabnik wrote:
>> Then isn't knowledge of the media type "out-of-band" information?
>
> This is why terminology is hard. :/
>
> Obviously, the processing rules are not included with the message
> itself. However, by pointing at a media type, we can say "If you
> understand text/html, you know _everything_ you need to know." Extra
> semantics outside of text/html would be out-of-band. Media type
> documentation is obviously not machine-readable, and a human will need
> to read them. But you can treat the Content-Type string as a pointer.
> A simple substitution.

I fail to see a distinction between documentation that a human must read and documentation a computer can use to navigate an API. It's still documentation, one requires human intelligence and ability to interpret the instructions and the other just happens to contain machine processable rules.

If I travel to France and need to speak to the waiter who can't speak English, I can use a phrase book or ask a friend to translate; both enable me to communicate with the waiter. Same is true of API documentation, written in English or machine processable in JSON.

> The practical point of this is to allow clients and servers to evolve
> independently of one another.

How does a service document disallow independent evolution? This is not session state nor is it data or content, these are the semantics of its processing rules. A service document enables evolution better than not having one. If processing rules can be changed by the service document and the service document is actively used to navigate the API then having a service document is better than having those same rules written and manually be interpreted by a programmer.

BTW, I would see the service document as basically the entry point to an API; a client would call it first to learn the API's navigation rules and then navigate the API using those rules.

> Out-of-band info requires the two parties to share information outside the protocol itself,
> which means they need to be in touch with each other. If everything is in the
> media type, clients and servers can be implemented without knowing
> about each other's details.

Respectively, I think you are viewing this "out-of-band" requirement by the letter and not the meaning and disregarding the context and the underlying goals.

-Mike

[Pat Cappelaere]

Steve,

Why would Media type documentation obviously not machine readable????

Let's look at Huddle http://code.google.com/p/huddle-apis/wiki/MediaType

application/vnd.huddle.data; version=1; describedby="https://schema.huddle.net/rng/folder.rnc"

This allows machines to understand what you mean and use it in a meaningful way in most cases (usually) without requiring out-of-band information (which is what we want)

:)

Now, I really want to know the value added of using custom media types on top of a customized type such as application/hal+siren on top of application/json and still end up with application/json that you need to describe to your users???

Wouldn't it be simpler to just describe your API in JSON schema and we are done?

What am I missing here?

Pat.

On Jul 8, 2012, at 10:55 AM, Steve Klabnik wrote:

[Mike Kelly]

On Sun, Jul 8, 2012 at 6:53 PM, Mike Schinkel <mi...@newclarity.net> wrote:
> On Jul 8, 2012, at 10:55 AM, Steve Klabnik wrote:
>>> Then isn't knowledge of the media type "out-of-band" information?
>>
>> This is why terminology is hard. :/
>>
>> Obviously, the processing rules are not included with the message
>> itself. However, by pointing at a media type, we can say "If you
>> understand text/html, you know _everything_ you need to know." Extra
>> semantics outside of text/html would be out-of-band. Media type
>> documentation is obviously not machine-readable, and a human will need
>> to read them. But you can treat the Content-Type string as a pointer.
>> A simple substitution.
>
> I fail to see a distinction between documentation that a human must read and documentation a computer can use to navigate an API. It's still documentation, one requires human intelligence and ability to interpret the instructions and the other just happens to contain machine processable rules.
>
> If I travel to France and need to speak to the waiter who can't speak English, I can use a phrase book or ask a friend to translate; both enable me to communicate with the waiter. Same is true of API documentation, written in English or machine processable in JSON.
>
>> The practical point of this is to allow clients and servers to evolve
>> independently of one another.
>
> How does a service document disallow independent evolution? This is not session state nor is it data or content, these are the semantics of its processing rules. A service document enables evolution better than not having one. If processing rules can be changed by the service document and the service document is actively used to navigate the API then having a service document is better than having those same rules written and manually be interpreted by a programmer.
>

Ok, I think the point that is being made is that referring to it as a
'service document' is a poor choice if what you are really talking
about is a normal resource that provides core navigational links and
is traversed over as part of the application itself. Talking about it
as a service document gives the impression that it is somehow special,
offers a full-description of a service, or contains information that
can be extracted and embedded into the client logic itself in a fixed
way.

Cheers,
M

[Mike Schinkel]

On Jul 8, 2012, at 2:30 PM, Mike Kelly wrote:
> Ok, I think the point that is being made is that referring to it as a
> 'service document' is a poor choice if what you are really talking
> about is a normal resource that provides core navigational links and
> is traversed over as part of the application itself.

Yes, I think you are correct. I've come to realize "service document" is a bad choice of name given the reaction to the name here on this list.

> Talking about it
> as a service document gives the impression that it is somehow special,
> offers a full-description of a service, or contains information that
> can be extracted and embedded into the client logic itself in a fixed
> way.

Problem is, I didn't know what else to call it. And what I was discussing is distinct from a resource that returns data, it is static for a given version of an API and defines what resources are valid and what knowledge is needed to properly navigate those resources (i.e. mapping of names like "order_id" to something like a JSON pointer to the location of the data within a representation returned by a specific resource.) Maybe we can collectively discover or coin a new name for this? Ideas?

-Mike
P.S. Also, could you see an extension to HAL that would allow it to be overlaid on existing APIs without requiring the existing APIs to be modified in any way, maybe using a /.well-known/hal.json resource[1] and URI Templates[2]?

[1] http://tools.ietf.org/html/rfc5785
[2] http://tools.ietf.org/html/rfc6570

[Pat Cappelaere]

On Jul 8, 2012, at 3:33 PM, Mike Schinkel wrote:

On Jul 8, 2012, at 2:30 PM, Mike Kelly wrote:

Ok, I think the point that is being made is that referring to it as a

'service document' is a poor choice if what you are really talking

about is a normal resource that provides core navigational links and

is traversed over as part of the application itself.

Yes, I think you are correct.  I've come to realize "service document" is a bad choice of name given the reaction to the

name here on this list.

The Atom Publishing Protocol calls this a service document: http://bitworking.org/projects/atom/rfc5023.html#appdocs

OpenSearch calls this an OpenSearch Description Document: http://www.opensearch.org/Home

Google calls this the API Discovery Service Document Resource:  https://developers.google.com/discovery/v1/reference#resource_discovery

We are talking about describing a web service… so a service document is pretty close :)

I really hope that we are not stuck on a name here.

Pat

[Mike Kelly]

On Sun, Jul 8, 2012 at 8:33 PM, Mike Schinkel <mi...@newclarity.net> wrote:
>
> P.S. Also, could you see an extension to HAL that would allow it to be overlaid on existing APIs without requiring the existing APIs to be modified in any way, maybe using a /.well-known/hal.json resource[1] and URI Templates[2]?
>
> [1] http://tools.ietf.org/html/rfc5785
> [2] http://tools.ietf.org/html/rfc6570
>

URI templates are already part of HAL, and I think a better approach
to [1] for overlaying this navigational HAL representation is to make
use of conneg and serve it from the entry-point(s) (e.g. the root URI)
using content negotiation

Cheers,
M

[Mike Schinkel]

The name "service document" doesn't bother me at all. 

But it *seems* to bother at least one other person here (I'm referring to Steve Klabnik and not Mike Kelly; Steve if I am assuming wrong I apologize, not trying to put words in your mouth.) It may also bother a lot of others[1] & [2] (for example), although what I'm talking about here is definitely not WADL.  So if a new name eased discussion by eliminating inadvertent prior negative bias them a new name might have benefit.

-Mike

[1] http://bitworking.org/news/193/Do-we-need-WADL

[2] http://bill.burkecentral.com/2009/05/21/to-wadl-or-not-to-wadl/

[Mike Schinkel]

On Jul 8, 2012, at 3:54 PM, Mike Kelly wrote:
> On Sun, Jul 8, 2012 at 8:33 PM, Mike Schinkel <mi...@newclarity.net> wrote:
>>
>> P.S. Also, could you see an extension to HAL that would allow it to be overlaid on existing APIs without requiring the existing APIs to be modified in any way, maybe using a /.well-known/hal.json resource[1] and URI Templates[2]?
>>
>> [1] http://tools.ietf.org/html/rfc5785
>> [2] http://tools.ietf.org/html/rfc6570
>>
>
> URI templates are already part of HAL,

Sorry, wasn't saying it wasn't, was instead trying to prescribe the idea.

> and I think a better approach to [1] for overlaying this navigational
> HAL representation is to make use of conneg and serve it from the
> entry-point(s) (e.g. the root URI) using content negotiation

Personal preference, I'm not a fan of conneg as it makes ad-hoc testing via browser difficult if not impossible.

Also, that approach cannot be layered on top of an existing API by a third party, i.e. if an API client wants to using a description document of their own authoring to navigate an API prior to the API itself adopting such an approach.

-Mike

[Glenn Block]

The draft is very new, don't feel bad :-)

[Glenn Block]

HTML does a have ubiquity on it's side however...I personally find it lacks in usability for an "API". You "can" use it and I can see the benefit of having something easily consumable by a browser. However when I read it / create it for an API I find myself continually squinting and there is a conceptual overhead. I have to map from HTML elements to a specific domain. Wheras if I look at a specific media type with a specific purpose, such as say vcard (Yes it does not support hypermedia, but regardless) then the semantics of the domain are expressed very richly and obviously. 

Also the implementation specifically on the consumption side is easier for vcard than say reading HTML. Yes i could use micro formats to embed a vcard in HTML but I'd only do that for a specify purpose where I am serving up say a web page that some extensions could also extract contact info from.

Lastly there is the leanness factor. Like it or not HTML is bloated. Yes there are tons of web servers that work with it etc, but that doesn't change the bloating factor. In the traditional model of request/response for HTML applications did far fewer requests than they do in the apps we are building today. Try seeing what happens if you rewrite something like say Gmail with passing HTML back and forth for Ajax calls. The bloat matters.

But for me the fact that HTML is primarily for layout in browsers and that I "can" bend it to be other things but that it is not inherently, is enough of a deal breaker.

My $.02

[mca]

Glenn:

Very interesting observations, well expressed. I've heard them before and I've not yet seen tangible examples of these burdens that HTML has over other hypermedia types.  It would help me if you could show me some comparison examples between your favorite hypermedia type and HTML.

Note that I am asking you to compare HTML to one or more hypermedia types, not HTML and vCard.

"I have to map from HTML elements to a specific domain. "

Yes, this happens all the time; every act of crafting a response involve mapping domain specifics to a message format. My Q here is this. what other _hypermedia_ type requires less "squinting" when you are mapping from "X" elements to a specific domain? A great help would be mapping domain specific data to your favorite _hypermedia_ type (NOT vcard!) and then mapping the same data to HTML. 

"Try seeing what happens if you rewrite something like say Gmail with passing HTML back and forth for Ajax calls. The bloat matters."

Again, I'd like to see comparisons of one of your favorite hypermedia types to HTML in passing back and forth Gmail data. Even a small example that shows how much less "bloat" there is in hypermedia format X over HTML will be helpful.

"But for me the fact that HTML is primarily for layout in browsers and that I "can" bend it to be other things but that it is not inherently, is enough of a deal breaker."

What elements in HTML that are layout oriented are you REQUIRED to use when using HTML for APIs? and which of these are better expressed using a hypermedia type of your choosing? IOW, an example that shows the _unavoidable_ layout-burden in an HTML API response vs the same response expressed in hypermedia type X and the elimination of that burden would be helpful.

Anything you can offer here would be most appreciated.

mca

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Glenn Block]

CIL

On Mon, Jul 9, 2012 at 10:51 AM, mca <m...@amundsen.com> wrote:

Glenn:

Very interesting observations, well expressed. I've heard them before and I've not yet seen tangible examples of these burdens that HTML has over other hypermedia types.  It would help me if you could show me some comparison examples between your favorite hypermedia type and HTML.

Note that I am asking you to compare HTML to one or more hypermedia types, not HTML and vCard.

"I have to map from HTML elements to a specific domain. "

Yes, this happens all the time; every act of crafting a response involve mapping domain specifics to a message format. My Q here is this. what other _hypermedia_ type requires less "squinting" when you are mapping from "X" elements to a specific domain? A great help would be mapping domain specific data to your favorite _hypermedia_ type (NOT vcard!) and then mapping the same data to HTML. 

OK, let's take Atom as an example. Atom has feeds and feed items. In HTML I would map one or more elements like <UL>, <LI> etc and say if you see a <UL> and there is a profile set, or there is a specific class i.e. "feed" then it is a feed. In Atom I would see elements in the document like "Feed" and "Entry" etc. Do you not agree that one is more intuitive than the other?

"Try seeing what happens if you rewrite something like say Gmail with passing HTML back and forth for Ajax calls. The bloat matters."

Again, I'd like to see comparisons of one of your favorite hypermedia types to HTML in passing back and forth Gmail data. Even a small example that shows how much less "bloat" there is in hypermedia format X over HTML will be helpful.

Here's something to illustrate the idea. In this case there is about a 30% difference in size between the HTML version and the JSON version. 

[

  {first:"Glenn", last: "Block", links:[{rel:"cn:about", href="."}]}, 

  {first:"Mike", last: "Amundsen", links:[{rel: "cn:about", href="."}]}

]

vs

<HTML>

<OL class="contacts">

  <LI>

    <P name="First">Glenn</P>

    <P name="Last">Block</P>

    <A rel="cn.about" href="."/>

  </LI>

  <LI>

    <P name="First">Mike</P>

    <P name="Last">Amundsen</P>

    <A rel="cn.about" href="."/>

  </LI>

</OL>

</HTML>

 

"But for me the fact that HTML is primarily for layout in browsers and that I "can" bend it to be other things but that it is not inherently, is enough of a deal breaker."

What elements in HTML that are layout oriented are you REQUIRED to use when using HTML for APIs? and which of these are better expressed using a hypermedia type of your choosing? IOW, an example that shows the _unavoidable_ layout-burden in an HTML API response vs the same response expressed in hypermedia type X and the elimination of that burden would be helpful.

Layout oriented may be the wrong term. HTML is a generic way to represent documents on the web. However the primary consumer of that document format which has drivne it's evolution IS a browser.

I'll use my example to indicate the friction.

The JSON example can simply be read in Javascript, and converted into an object. I can then navigate the graph of the object simple accessing named properties. The one exception are the links which are a generic collection.

In the case of the HTML example I either parse manually or I read it into a DOM object. If I go with a DOM object I then have to read a recursive set of nested collections looking at the class/rel to pull out the information.

That to me is an example of the friction that I am talking about.

[Glenn Block]

Actually my <P> should actually be <INPUT> boes most likely which would further bloat the size on a large list.

[Glenn Block]

Back to my example I will argue that the first is more intention revealing / intuitive in terms of the processing of it's information by a human as it is designed with a very specific purpose, that is contacts. The second which is using HTML is also design with a very specific purpose, information exchange, but that encompasses an extremely broad domain.

[Greg Brail]

All the hypermedia JSON examples that I see really like to use this pattern, which is I assume borrowed from HTML:  

"links": [

    { "rel": "self", "href": "http://api.x.io/orders/42" },

    { "rel": "previous", "href": "http://api.x.io/orders/41" },

    { "rel": "next", "href": "http://api.x.io/orders/43" }

  ]

Given how often a truly hypermedia-aware API client would theoretically have to iterate through links to get work done, that means that the client has to iterate through the array of links, find the one with the right "rel", and then fetch the "href" element, and handle all the special cases of those things not being present. Admittedly this is simple using JSONPath, just as it was simple using XPath with XML, but I kind of thought that everyone liked JSON better these days because it's easier for typical clients to parse and didn't require a "stack" on the client side just to handle basic stuff.

What is the advantage of the pattern above over:

"links": [

  "self": "http://api.x.io/orders/42",

  "previous": "http://api.x.io/orders/41",

  "next": "http://api.x.io/orders/43"

]

because in this second example, all a JavaScript client needs in order to follow a link is a simple property reference.

On Fri, Jul 6, 2012 at 5:27 PM, Kevin Swiber <ksw...@gmail.com> wrote:

Howdy, folks!

I recently started piecing together a specification of a hypermedia type.  I'd love to get your thoughts and ideas for improvement.  Many of your conversations here and your related work have been an inspiration.

First draft: https://github.com/kevinswiber/siren

I've been working on a Hypermedia API that has gone through several iterations.  Siren is a result of my latest efforts.

Some of the goals have been to represent, in a meaningful way:

- Communication of current state

- Embedded resources

- Embedded resource links

- Navigation links

- Actions

I developed Siren with an Entity concept, defined as being URI addressable and containing sub-entities, actions, and navigational links.

I have been focusing on JSON Siren as the first implementation, because it fits in nicely with my technology stack (Node.js, CouchDB, Web frontend).

Head on over to https://github.com/kevinswiber/siren to take a peek!  I'm happy to have discussions, and I'm also open to GitHub Issues and Pull Requests.

Thanks!

--
Kevin Swiber

Projects: https://github.com/kevinswiber
Twitter: @kevinswiber

--
Gregory Brail  |  Technology  |  Apigee

[Glenn Block]

"My Q here is this. what other _hypermedia_ type requires less "squinting" when you are mapping from "X" elements to a specific domain?"

HAL requires far less squinting though true it does require some. However I am not the uber fan of HAL as the one true media type for the same reason.

[Glenn Block]

One advantage is you can multiple values with the same rel, there is nothing stopping that. In the case you have listed that won't work as the key must be unique.

[mca]

Glenn:

great examples! thanks.

some feedback:

Mapping domain specifics

atom:feed vs <ul class="feed">...</ul> is definitely exposing some "indirection" between atom and HTML, but I assert this is a "false" example. Atom is most often used to express _domain_specific_ items like users, companies, invoices, etc. Doing this in Atom works the same as in HTML (via a decorator on an element, etc.).  That's why I asked about  a domain specific example.

Bloat:

yes, HTML is definitely "beefier", i do observe that your exmples do not see quite "comparstive" and i've updated them here. in fact, i think the "byte" count for the HTML example goes up higher in my example<g>

 

{contacts :

[

  {first:"Glenn", last: "Block", links:[{rel:"cn:about", href="."}]}, 

  {first:"Mike", last: "Amundsen", links:[{rel: "cn:about", href="."}]}

]

vs

<div class="contacts">

  <form href="...">

    <input name="First" value="Glenn" />

    <input name="Last" value="Block"/>

    <A rel="cn.about" href="."/>

  </form>

  <form href="...">

    <input name="First" value="Mike" />

    <input name="Last" value"Amundsen" />

    <A rel="cn.about" href="."/>

  </form>

</div>

Layout-Dependence (now Friction)

First, I think it wise to drop the "layout" claim on HTML. many message formats (including JSON) can be treated as rendering information.  That HTML _started_ like that is hardly a requirement that all HTML messages be treated that way.

As to "friction" you seem to be citing tooling/parsing differences here. IOW, that the JSON "DOM" is less rich, more direct than the HTML DOM. I agree w/ this. However, this is, i think, a repeat of the "bloat" and "mapping" arguments you've already put forth.

I think you are right to say you prefer JSON over HTML to express objects since it's more terse and HTML does not express objects. I think you're right to say you prefer Atom over HTML to express feeds since HTML is not designed to express feeds.

One of the items that did not appear in your examples is expressing hypermedia affordances. IME, this is the area where HTML excels since so many are already "built-in" to the design and other, more terse/object-friendly media types lack them (XML, JSON, CSV, etc.).

+1.859.757.1449
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

[Mike Kelly]

On Mon, Jul 9, 2012 at 4:44 AM, Greg Brail <gr...@apigee.com> wrote:
> All the hypermedia JSON examples that I see really like to use this pattern,
> which is I assume borrowed from HTML:
>
> "links": [
> { "rel": "self", "href": "http://api.x.io/orders/42" },
> { "rel": "previous", "href": "http://api.x.io/orders/41" },
> { "rel": "next", "href": "http://api.x.io/orders/43" }
> ]
>
> Given how often a truly hypermedia-aware API client would theoretically have
> to iterate through links to get work done, that means that the client has to
> iterate through the array of links, find the one with the right "rel", and
> then fetch the "href" element, and handle all the special cases of those
> things not being present. Admittedly this is simple using JSONPath, just as
> it was simple using XPath with XML, but I kind of thought that everyone
> liked JSON better these days because it's easier for typical clients to
> parse and didn't require a "stack" on the client side just to handle basic
> stuff.
>
> What is the advantage of the pattern above over:
>
> "links": [
> "self": "http://api.x.io/orders/42",
> "previous": "http://api.x.io/orders/41",
> "next": "http://api.x.io/orders/43"
> ]
>
> because in this second example, all a JavaScript client needs in order to
> follow a link is a simple property reference.

Yes, the latter makes far better use of JSON and is how hal+json works
(links are objects not strings):

{
"_links": {
"user": { "href": "/users/123", "title": "Bob" }
}
}

Cheers,
M

[Kevin Swiber]

Hey Greg,

Siren does not use the rel value to access the link due to the requirement I impose that links should support multiple rel values.

While it is not illegal in JSON to have a space in the property name, it's still awkward to parse (by humans and machines alike).

Given the requirement of links supporting multiple rel values, how would you like to see this designed?

Thanks.

[Mike Kelly]

On Mon, Jul 9, 2012 at 11:43 AM, Kevin Swiber <ksw...@gmail.com> wrote:
> Hey Greg,
>
> Siren does not use the rel value to access the link due to the requirement I
> impose that links should support multiple rel values.
>
> While it is not illegal in JSON to have a space in the property name, it's
> still awkward to parse (by humans and machines alike).

A JSON name is a string, so there is no issue for a machine parsing a
space separated list of rels as a JSON name.

The issue is with parsing multiple rels and providing an API for
selection; which is a challenge regardless of whether you are using
the named link approach vs the array-search approach.

Cheers,
M

[Kevin Swiber]

"Machines" in this context is really referring to existing JSON parsers.  I should have made that clear.  JSON parsers often attempt to generate object structures in <your language of choice>.  Putting a space in a property name will get hypermedia implementors into the weeds of how their parser of choice handles spaces.  (For some languages, this is fine as a valid property name.  For others, there has to be a workaround.)  While an API consumer may be perfectly fine handling this scenario, I'd prefer to not force it on folks right out of the box.  (To be fair, there are other characters that may cause issues, as well.)

Compounding the problem with "named links" is that link rel values may not be unique in a collection of links.  Think of a family tree representation that contains 2 "brother" links on the same element.  The way a JSON parser handles two values of the same property name will differ across parsers.

And to me, that contention is there because the rel values and links are not a 1-to-1 metaphor that maps cleanly onto a key-value pair representation.  To get to that state requires setting up boundaries and restrictions on how implementors use the media type.  There's nothing wrong with that as long as it's in line with the goals, requirements, and vision.  I think this piece would break a requirement in Siren, however.

Regards,

[Greg Brail]

Yeah, that makes sense especially since my example is not syntactically correct ;-)

[Greg Brail]

Why would a link have multiple rels? What would that mean?

[Kevin Swiber]

Good question.  It might help to see an example.  I'll pull one from Mike Amundsen's work on RFC 6573, The Item and Collection Link Relations [1].

From Section 2.2:

   It should also be noted that the same link might represent an 'item'
   in one collection as well as a 'collection' itself.  In this case,
   both link relation values can be applied to the same link:

   Link: <...>; rel="collection item";
         title="A Review of Issac Asimov's Collected Works - Vol. I"

Of course, there could also be any combination of rel values coming from extension relation types [2].  For example, a list of contacts could contain a contact record that is both a "customer" and a "vendor."  In that scenario, the record may contain a link rel value of "http://x.io/rels/customer http://x.io/rels/vendor".  (Note that "x.io" is just an example hostname for brevity.)

[1] http://tools.ietf.org/html/rfc6573
[2] http://tools.ietf.org/html/rfc5988#section-4.2

[Mike Kelly]

On Mon, Jul 9, 2012 at 1:22 PM, Kevin Swiber <ksw...@gmail.com> wrote:
>
>
> On Mon, Jul 9, 2012 at 6:57 AM, Mike Kelly <mikeke...@gmail.com> wrote:
>>
>> On Mon, Jul 9, 2012 at 11:43 AM, Kevin Swiber <ksw...@gmail.com> wrote:
>> > Hey Greg,
>> >
>> > Siren does not use the rel value to access the link due to the
>> > requirement I
>> > impose that links should support multiple rel values.
>> >
>> > While it is not illegal in JSON to have a space in the property name,
>> > it's
>> > still awkward to parse (by humans and machines alike).
>>
>> A JSON name is a string, so there is no issue for a machine parsing a
>> space separated list of rels as a JSON name.
>>
>> The issue is with parsing multiple rels and providing an API for
>> selection; which is a challenge regardless of whether you are using
>> the named link approach vs the array-search approach.
>
>
> "Machines" in this context is really referring to existing JSON parsers. I
> should have made that clear. JSON parsers often attempt to generate object
> structures in <your language of choice>. Putting a space in a property name
> will get hypermedia implementors into the weeds of how their parser of
> choice handles spaces. (For some languages, this is fine as a valid
> property name. For others, there has to be a workaround.) While an API
> consumer may be perfectly fine handling this scenario, I'd prefer to not
> force it on folks right out of the box. (To be fair, there are other
> characters that may cause issues, as well.)

Do you have any specific examples of libraries that this presents an issue for?

> Compounding the problem with "named links" is that link rel values may not
> be unique in a collection of links. Think of a family tree representation
> that contains 2 "brother" links on the same element. The way a JSON parser
> handles two values of the same property name will differ across parsers.

Two properties with the same name would be invalid JSON, if you have
multiple links in HAL then you use an array of link objects:

{
"_links": {
"brother": [{
"href": "/people/bob"
},{
"href: "/people/ted"
}]
}


Cheers,
M

[Kevin Swiber]

On Mon, Jul 9, 2012 at 11:02 AM, Mike Kelly <mikeke...@gmail.com> wrote:

On Mon, Jul 9, 2012 at 1:22 PM, Kevin Swiber <ksw...@gmail.com> wrote: 

 

> "Machines" in this context is really referring to existing JSON parsers.  I
> should have made that clear.  JSON parsers often attempt to generate object
> structures in <your language of choice>.  Putting a space in a property name
> will get hypermedia implementors into the weeds of how their parser of
> choice handles spaces.  (For some languages, this is fine as a valid
> property name.  For others, there has to be a workaround.)  While an API
> consumer may be perfectly fine handling this scenario, I'd prefer to not
> force it on folks right out of the box.  (To be fair, there are other
> characters that may cause issues, as well.)

Do you have any specific examples of libraries that this presents an issue for?

I can't fire up a new dev environment at the moment, but any .NET JSON deserializer (Newtonsoft is a popular one) would need custom parsing logic to make this happen.  (It's not impossible; it's just more work.)  I would probably deserialize the links into a collection of objects and set the rel value to a C# property value to get around this issue.  (Again, there are many characters in a link rel value that would not be a valid property in C#.)  Making links a collection of objects that contain the rel value in a JSON property value just cuts to the chase.

 

> Compounding the problem with "named links" is that link rel values may not
> be unique in a collection of links.  Think of a family tree representation
> that contains 2 "brother" links on the same element.  The way a JSON parser
> handles two values of the same property name will differ across parsers.

Two properties with the same name would be invalid JSON, if you have
multiple links in HAL then you use an array of link objects:

{
  "_links": {
    "brother": [{
      "href": "/people/bob"
    },{
      "href: "/people/ted"
    }]
}

Having two of the same property names in a JSON object is not invalid [1].  I actually had to double-check this after you called me out.  :)  

This isn't really about HAL, however.  Perhaps HAL is well-defined on this issue, including all the exceptions and gotcha's that might come from the rel-as-a-property-name approach.  If not, I'd be happy to offer suggestions for additions to the draft.  If it is well-documented, I'd love to learn from it.  I think Siren should handle these exceptions, as well, and at this point, I'm only seeing more evidence that the rel-as-a-property-value route is the easiest way to do that, for both API producers and consumers.

[1] Section 2.2 of http://tools.ietf.org/html/rfc4627

[Mike Kelly]

Right, but you're "cutting to the chase" based two edge cases coming
together, specifically:

* you need multiple relations, and..
* there's a client that needs to use a buggy JSON API

if neither of those are the case then this is a non-issue.

In my opinion, a clean interface for the vast majority that makes the
most of the base media type, is more important than trying to account
for every possible edge-case that could occur with poorly implemented
libraries. That approach is basically the antithesis of the design
goals of HAL.. I've tried to create something that will make hypertext
accessible to 'normal' people by purposefully rejecting the temptation
to overcomplicate it with features most people don't need. I just want
to help people doing hypermedia, I'm not trying to prove how many
hypermedia "affordances" I can dream up. Fwiw, I have dreamt up many -
just saying! ;D

Cheers,
M

[Kevin Swiber]

You misunderstand.  There is no "buggy JSON API" here, and I'm not sure where you're getting that idea.  I'm representing the information in a way that supports a broad set of use cases, which allows an API to start with the most simple use case and grow into a more complex one over time without breaking the client.  That's the goal.  Easing the client implementation across multiple platforms and languages is a by-product and a supporting reason to keep the design.

I'm not saying other options are not as good in general.  Specifically, I'm asserting that "named links" are not as good at supporting the growth of complexity over time nor are they good at supporting currently complex implementations.  It seems you agree with this, and I agree that "named links" are a simpler option for the most basic (and likely most common) use case.

I think we are on the same page.  :)

[Jon Moore]

On Sun, Jul 8, 2012 at 9:33 AM, Pat Cappelaere <cappe...@gmail.com> wrote:

I have a lot of respect for Jon Moore and HTML but… any decent HTML page these days will have a lot of javascript, jquery callbacks, social media tracking and so on…  That's fine to render a view targeted to a human.  But, if you output a representation for a machine, it may behoove you to use something more appropriate.  This could be a bare bone HTML but then why not use XML or JSON?  HTML scraping is not a best practice.  Simpler may be better.

As for why not XML or JSON--they don't have native hypermedia support. HTML scraping is brittle (I know because I've written them before) primarily because most markup doesn't have semantic markers on it. HTML5 microdata, RDFa, etc. are changing that, and you just have to look to things like Google's rich snippets testing tool to see that this is possible. 

For me, there's also a benefit of serendipitous reuse. If a human happens to end up looking at my "API" in a browser, she can potentially make use of it (although it won't be the slickest UI out there). Similarly, a programmatic client that can grok HTML can potentially (and that's a big caveat) make sense of regular websites. But if my programmatic client can't grok HTML, all that web content is out of reach.

 

Would we be in agreement and say:
- Web users are not just humans but more and more machines.

I would agree we need to serve both human and programmatic clients.

 

- Hypermedia API is here to stay.  No question about it.  Machines need to be able to follow links AND understand what they are following.  That's the real challenge.

Agreed. I've found that HTML, link relations, and microdata provide plenty of foothold for this.

 

- m2m rendering is a must.  Use what is appropriate for your needs but document it in a way that can be used m2m as well.  I do not see a way around some kind of a service document and simple schema.  We need to use it for our docs and internal validation. It has to stay current as the app evolves over time.

So far that's not our experience; we can serve HTML representations that are intended for programmatic clients but which are also usable by (human + browser) clients. We primarily document microdata schema (although these are, in reality, quite relaxed schema), link relations, and form @classes and input @names.

Jon 

........
Jon Moore

[Jon Moore]

On Sun, Jul 8, 2012 at 1:12 PM, Mike Schinkel <mi...@newclarity.net> wrote:

Four things:

1.) HTML is notoriously difficult to parse whereas JSON is absolutely trivial to parse in most web languages/platforms.  XHTML is easier, but still no where near as easy as JSON is some web languages/platforms.

And yet there are many implementations (browser guts, specifically) that can parse HTML. There are HTML parsers in Python, Ruby, Javascript, Perl, PHP, C, Java, etc. Is writing "JSON.parse(s)" really that much easier than "myHTMLLibrary.parse(s)"? I will agree that on most platforms manipulation of the parsed result is often unwieldy, especially compared to using parsed JSON in dynamically typed languages where it can be rendered as "native" data structures. However, there's something even easier: JQuery. I think if we ported JQuery selector syntax into other languages, we'd find lots of people who have no problem whatsoever manipulating HTML.

 

2.) X/HTML response representations are bloated compared to their JSON equivalent and that's an issue for mobile clients.

This issue goes away quite a bit with the use of compression. You can tell the difference in gzipped formats at small response bodies (where, to be honest, the difference doesn't matter as much), but as the response sizes grow, the compression algorithm helps you more and more, as it asymptotically approaches the actual information theoretical content. 

3.) HTML didn't really apply here as you referred to my post about a proof-of-concept "service document" that defined rules for processing Twitter's JSON based API, and 

Fair point. Sorry for threadjacking.

 

4.) Ironically the difficult part of writing the service document was not the display to the use but instead presenting the actions that could be performed by the API. I would have had the exact same difficulty if I had encoded the service document in HTML.

No arguments here--HTML would have difficulty describing a pre-existing API that doesn't accept requests that could be driven by links or forms. On the other hand, HTML is great for describing an HTML-based API.

[Pat Cappelaere]

Jon,

Now I am confused.

Are you saying that, based on your experience,  we ought to consider HTML as the one true media-type?????

or are you saying that it worked in your limited application but might not work not for majority of larger applications?

Pat.

[Jon Moore]

On Sun, Jul 8, 2012 at 10:36 PM, Glenn Block <glenn...@gmail.com> wrote:

HTML does a have ubiquity on it's side however...I personally find it lacks in usability for an "API". You "can" use it and I can see the benefit of having something easily consumable by a browser. However when I read it / create it for an API I find myself continually squinting and there is a conceptual overhead. I have to map from HTML elements to a specific domain. Wheras if I look at a specific media type with a specific purpose, such as say vcard (Yes it does not support hypermedia, but regardless) then the semantics of the domain are expressed very richly and obviously. 

I think you're getting at the difference between a domain-agnostic media type (HTML) and a domain-specific media type (vcard), and agree with the pros/cons you've laid out. I often find myself working in evolving domains with new concepts and functionality, and in these settings standardizing a domain-specific media type would be premature. It's also quite common for me to run into situations where it's hard to distinguish domain boundaries, or where I want to include concepts from multiple domains at the same time, and so a domain-agnostic media type works better for me here. But yes, if I'm dealing with contact information, I'd want to use vcard. :) 

 

Lastly there is the leanness factor. Like it or not HTML is bloated. Yes there are tons of web servers that work with it etc, but that doesn't change the bloating factor. In the traditional model of request/response for HTML applications did far fewer requests than they do in the apps we are building today. Try seeing what happens if you rewrite something like say Gmail with passing HTML back and forth for Ajax calls. The bloat matters.

I'll counter your "bloat" with "compression". But I'll follow up with some experimental data shortly.

 

But for me the fact that HTML is primarily for layout in browsers and that I "can" bend it to be other things but that it is not inherently, is enough of a deal breaker.

I'll disagree that HTML is primarily for layout--it's for conveying semantic information. CSS is for layout, and to be honest, the major layout-oriented element in HTML is <div>, and that's primarily to give the CSS a nice rectangle to hold on to. I found a link to a screenshot, purportedly, of Tim Berners-Lee's HTML editor. Please find a graphic designer who will say he was concerned with layout.

Is the confusion here that people think if you want to serve HTML up in your API that it has to look like the HTML served on modern websites (with all the bells and whistles)? It doesn't.

Jon

........
Jon Moore

[Jon Moore]

On Sun, Jul 8, 2012 at 11:35 PM, Glenn Block <glenn...@gmail.com> wrote:

I'll use my example to indicate the friction.

The JSON example can simply be read in Javascript, and converted into an object. I can then navigate the graph of the object simple accessing named properties. The one exception are the links which are a generic collection.

In the case of the HTML example I either parse manually or I read it into a DOM object. If I go with a DOM object I then have to read a recursive set of nested collections looking at the class/rel to pull out the information.

Or you could use something like JQuery and access the DOM quite elegantly.

 

That to me is an example of the friction that I am talking about.

Jon

-- 

........
Jon Moore

[Mike Schinkel]

And yet there are many implementations (browser guts, specifically) that can parse HTML. There are HTML parsers in Python, Ruby, Javascript, Perl, PHP, C, Java, etc. Is writing "JSON.parse(s)" really that much easier than "myHTMLLibrary.parse(s)"?

1000 times easier :). Most importantly, using JSON eliminates a dependency that using HTML would require.

And for that dependency I have to worry about if the library will be supported in the future, if killer bugs will be fixed. Plus the objects that result from json_decode($json) are much simpler to process than the ones from file_get_html($json_file)  (looking at: http://simplehtmldom.sourceforge.net/ for PHP.)

For Javascript, the example is from a 2008 blog post with closed comments and numerous reported bugs, not from an active open-source repository, and there don't appear to be any better options.  Plus, I'd have to include it too, rather than just $.getJSON() in jQuery.

Take PHP & Javascript and that's about 1/2 of web development (give or take 25%.)

Using HTML, which has all its foibles that work well for browsers, is just not a good pick for machine-to-machine web services. IMO anyway.

I think if we ported JQuery selector syntax into other languages, we'd find lots of people who have no problem whatsoever manipulating HTML.

Operative word: "IF"   Right?   Are there any known plans for this?

You can tell the difference in gzipped formats at small response bodies (where, to be honest, the difference doesn't matter as much), but as the response sizes grow, the compression algorithm helps you more and more, as it asymptotically approaches the actual information theoretical content. 

Since I don't do mobile development I don't worry about these things but I'd be really interested to hear what an active mobile developer who deals with high usage apps or apps with large amounts of data transfer would have to say as a counter to what an advocate for HTML would have to say. :)  Anyone?

-Mike 

[Glenn Block]

"Or you could use something like JQuery and access the DOM quite elegantly.:

If I am running in a browser / or something like node maybe, but there are still PLENTY of native apps ;-)

[Jon Moore]

On Mon, Jul 9, 2012 at 11:41 PM, Glenn Block <glenn...@gmail.com> wrote:

"Or you could use something like JQuery and access the DOM quite elegantly.:

If I am running in a browser / or something like node maybe, but there are still PLENTY of native apps ;-)

Actually, I did a little looking around on this, and there are ports of the JQuery selector syntax to Java and Python at least. In my opinion, it'd be a lot easier to do an open source port like this to your platform of choice, where you could attract a larger body of potential contributors (lots of people would like to manipulate HTML from language X, even if it's not for APIs) than it would be to garner support around a custom media type. But that's my opinion, which is why I'm working in this direction. :)

I also recognize this is an opinion, and will totally root for folks that want to develop custom media types like Siren or HAL or even domain-specific JSON formats. If those gain traction, I'll happily switch to using them. I'm not really trying to convince anyone else to use HTML, just trying to explain why it works for me. It might not work for you, because you may have different goals or constraints.

Jon

--
........
Jon Moore