Guidance/Patterns/Standards for Hypermedia vs. URL Construction?

[Mike Schinkel]

Hi all,

I've been interested in RESTful architectures since around 2005 but I've yet to find any really good examples in the wild of using hypermedia for RESTful APIs.

If I understand correctly the value of hypermedia is to enable a robust architecture that is resilient to changes in URLs, correct? But how is it best done? For example, how should links be returned? As headers? In the content response? 

How should hyperlinks be identified; as name/value pairs?  Are there any standard for naming, are there any standard data structures that can identify the list of links?

And how are expected variables identified, and how are their values specified?

Are there any attempts to standardize these for horizontal use-cases?  How about within specific use-cases,  as might have been done here?:

http://networkedhelpdesk.org/api/

One benefit for standardization is to enable better tools to emerge, but it seems every implementation is a custom for the specific user's use-case? Or have I missed tools that are emerging? If not, are there at least some best practices documents that have emerged to be seminal?

Thanks in advance for your thoughts.

-Mike

[Kin Lane]

I'd recommend taking a look at the Hypermedia approach of ElasticPath - http://www.elasticpath.com/

I'm still crash coursing all their stuff...so I can't speak to the best practices yet.  

But the 2 demos I've had, their HATEOS implementation was well thought out...and they were able to communicate it really well.

[Mike Schinkel]

On Mar 22, 2012, at 5:44 PM, Kin Lane wrote:

I'd recommend taking a look at the Hypermedia approach of ElasticPath - http://www.elasticpath.com/

But the 2 demos I've had, their HATEOS implementation was well thought out...and they were able to communicate it really well.

Thanks Kin.  Yes, the ElasticPath implementation does appear to be well-thought out; too bad they bury their docs behind a login wall and then make it very difficult once logged in to find docs that show an example.

But I'm surprised; I feared my question might erupt a heated debate yet you were the only one who answered.  Are there really no other good, public and more well-known examples of HATEOAS in the wild?

Last week one of my colleagues and I worked on the architecture for a RESTful client class for our use-case (WordPress integration plugins for our SaaS vendor clients.)  We studied the ElasticPath implementation and tried to architect a client that would interact with SaaS APIs as if they used HATEOAS. We discussed abstracting the URL construction required by most public REST-like APIs.  We thought that would allow our client to support HATEOAS when and if we came across client APIs that used HATEOAS.

However, we eventually realized that processing APIs using HATEOAS adds a level of abstraction and thus so much complexity that we gave up on that approach. We decided to just do URL construction which is all that most (almost all?) public REST-like APIs prescribe. If there were a standard method for HATEOAS by content type blessed by a recommended RFC, i.e. something like the ElasticPath approach that using "self" and "links" in JSON, then maybe it would be worth implementing HATEOAS into our client.  But as is, there seems that there would be no identifiable ROI for an investment in development of HATEOAS for our client for our use-cases. 

And that's a shame considering how many people I've seen publicly state that HATEOAS is critical for a proper RESTful architecture. But until there's a standard way to do it via HTTP, I don't see how this is going to change?  If HATEOAS is so crucial, why has there been so few publicly posted examples?  Why have there not been many (any?) open source implementations?  Why are so few companies and /or developers embracing HATEOAS in the wild?

OTOH, if my analysis is off-base, please do help me to see it differently. I would very much like that.

-Mike

[Kin Lane]

Hey Mike,

Yeah, I"m still working on digesting the ElasticPath implementation....Once I hack on more it will sink in deeper.

I think e-commerce is a universal enough concept, that I can use them to tell the story of why HATEOAS is important.

I can't 100% say why HATEOAS is so considered as being so crucial, yet so few examples to showcase.  I think it similar to what you said, the overhead in both implementing and explaining to your devs just doesn't make sense?  IDK.  I'd like to understand more.

I believe it has huge potential, but the story has to be told right.  I think ElasticPath + e-commerce is perfect, because of the known flows product to cart to checkout, etc.  And the common business logic....but it has to be really shown to make devs life easier, meet the biz requirements of API owner, and provide ROI....otherwise it will leave bad taste in the mouth.

Even though the success of RESTful APIs is definitive, it seems precarious and sensitive sometimes.  Everyone seems afraid to make the same mistakes as we made with SOA / Web Services and be to restrictive?  IDK...again...need to understand more.

I'll have more stories on ProgrammableWeb and API Evangelist on this as I wrap my mind around more.

Thanks,

Kin Lane

[Mike Schinkel]

On Mar 26, 2012, at 1:23 AM, Kin Lane wrote:

I think e-commerce is a universal enough concept, that I can use them to tell the story of why HATEOAS is important.

I believe it has huge potential, but the story has to be told right.  I think ElasticPath + e-commerce is perfect, because of the known flows product to cart to checkout, etc.  And the common business logic....but it has to be really shown to make devs life easier, meet the biz requirements of API owner, and provide ROI....otherwise it will leave bad taste in the mouth.

There are two levels of standardization that could take place; generic HATEOAS JSON formats so that generic frameworks could be built, and then an eCommerce standard built on top of a generic formats standard.  If the latter was attempted to be standardized before the former, it could result in incompatible approaches across business use-cases, which would probably be worse instead of better.

I can't 100% say why HATEOAS is so considered as being so crucial, yet so few examples to showcase.  I think it similar to what you said, the overhead in both implementing and explaining to your devs just doesn't make sense?  IDK.  I'd like to understand more.

I understand why it's important; i.e. using HATEOAS makes a system more resilient to URL change, I'm just not sure the benefit it really all that dramatic considering HATEOAS just punts the ball downstream (to mix metaphors.)  Sure the URLs are no longer a breaking point but then the meta data format in the response becomes an issue.  And I'm not sure if that's actually better.  At least that's what I'm currently seeing.

I'll have more stories on ProgrammableWeb and API Evangelist on this as I wrap my mind around more.

Subscribed. Looking forward to them.

-Mike

[Jørn Wildt]

Regarding HATEOS ...

If you create an API and document the URLs in the style of Twitter (https://dev.twitter.com/docs/api) and many others then HATEOS becomes irrelevant. Why? Because you have, up front, instructed the client about any and all URLs it will need. There is nothing left to discover at runtime.

If you create a media-type then there is no way for you to inform the client about the URLs up front. Take for instance the ATOM media-type (application/atom+xml) - you will find lots of link-relations in it, but not a single word about specific URLs. Now everything has to be discovered at runtime. That's when you need HATOES - its an absolute necessity when working with media-types.

By creating a media-type with associated link-rels you decouple application semantics and URL structure completely. A media-type allows you to reuse your application on other systems and servers. HATEOS is not just about adding flexibility to a specific API - it is about being able to reuse the application in completely different places. You cannot do that with the Twitter API - but you can do it with ATOM and other media-types.

As Eric Wilde wrote on Twitter: "new #REST rule: no, you cannot play with your URIs until you have designed all of your media and link relation types."

/Jørn

[Mike Schinkel]

I've yet to hear anyone make that distinction.  That's hugely helpful; thanks.

As an aside regarding ATOM, it's a rather application-specific media-type, and rather complex to implement, no?   I remember when it was being developed; there were untold man-hours invested in getting it to its current state. Maybe that's why there are so few other public examples, because of the effort required for definition?

-Mike

[Jørn Wildt]

I know nothing about the development process of ATOM. But I would assume it took so long time because it had to encompass many different scenarios and had a lot of different interested parties. That makes for lots of discussions.

But a more vendor specific media-type would not have those issues - it shouldn't take longer to define your application in terms of a media-type than it would take to define in terms of URLs.

/Jørn

[Mike Schinkel]

On Mar 26, 2012, at 3:04 AM, Jørn Wildt wrote:

I know nothing about the development process of ATOM. But I would assume it took so long time because it had to encompass many different scenarios and had a lot of different interested parties. That makes for lots of discussions.

But a more vendor specific media-type would not have those issues - it shouldn't take longer to define your application in terms of a media-type than it would take to define in terms of URLs.

It would be helpful if there were a lower level generic foundation for HATEOAS so that each vendor specific implementation would not have to reinvent the wheel.  That would enable significantly more reuse than vendor-specific implementations that are developed from the ground-up, and didn't you say that major benefit of HATEOAS?

I guess what I'm saying is I believe it's incumbent upon those who advocate HATEOAS as being critical to recognize the need for a shared standard for processing HATEOAS and then to develop such a standard so that each and every implementor does not have to reinvent the wheel.

-Mike

[Jørn Wildt]

You are absolutely right - and that leads to the discussion about "what's in a media-type?". Feel free to chime in on this discussion: https://groups.google.com/d/msg/hypermedia-web/V0hcWwQet3I/KTjnUOo1WHsJ

When talking about "reuse" we have to be careful about what we want to reuse. There's reuse of media-types that defines links, forms and other hyper-media controls ... and then there's reuse of application/domain semantics.

> It would be helpful if there were a lower level generic foundation for HATEOAS so that each vendor specific implementation would not have to reinvent the wheel

There is: HTML and HAL (to name those I know of). HTML is not normally considered useful for machine-to-machine APIs, but it does include links and forms that allows you to do a lot of stuff - and then you can encode your data with some sort of micro format.

Assuming you accept the idea of using HTML for machine APIs: now you have all (many of) the low level HATEOS constructions you need (anchors, forms, links) - any vendor can use that format to encode machine readable APIs for something like Twitter/Last.fm/Salesforce/whatever. But now the question changes - if the client receives a generic format like HTML/HAL how does it then discover the domain specific semantics?

Your feedback would be much appreciated here: https://groups.google.com/d/msg/hypermedia-web/V0hcWwQet3I/KTjnUOo1WHsJ

/Jørn

[Duncan Cragg]

 

It would be helpful if there were a lower level generic foundation for HATEOAS so that each vendor specific implementation would not have to reinvent the wheel.  That would enable significantly more reuse than vendor-specific implementations that are developed from the ground-up, and didn't you say that major benefit of HATEOAS?

I guess what I'm saying is I believe it's incumbent upon those who advocate HATEOAS as being critical to recognize the need for a shared standard for processing HATEOAS and then to develop such a standard so that each and every implementor does not have to reinvent the wheel.

There is a growing list of approaches that are in this approximate area: 

• JSON-LD
  
• HAL
  
• Collection+JSON
  
• OData/JSON
  
• JSON-{Schema,Reference,Pointer,Patch} (Kris Zyp)
  
• “A Convention for HTTP Access to JSON Resources”
  
• My own: 'The Object Network'
  

My Object Network approach is as different as any of them are to each other.

I have a couple of articles up about it on InfoQ and on ProgrammableWeb:

http://www.infoq.com/news/2012/02/the-object-network

http://blog.programmableweb.com/2012/02/13/the-object-network-making-mashups-easy/

Also: http://the-object.net

Cheers!

Duncan