Link Headers vs links in response body

[Ryan Connolly]

Hello API-Crafters,

This is my first post to the group and have been enjoying the excellent conversations here.  There is one topic that I can't seem to find a straight forward answer to and that is the one regarding how to best include Link rels in responses to address HATEOAS.  I've seen arguments for using Link Headers and arguments for including links in response bodies and I'm leaning towards the Link Headers.  For a little background, I would at the minimum want to support JSON, XML, and HTML.  Obviously links would be included in the response body for HTML responses but I'm struggling with what is best for the other two media-types.  I'm aware of ATOM links and HAL but through my experience of consuming APIs, it doesn't seem like anyone can agree on a standard approach to linking for such media-types.  Any insight into this topic from you crafters would be greatly appreciated.

-Ryan

[Mike Amundsen]

Ryan:

welcome to the list; it's a good group.

I place links within the body of the response whenever possible. I treat the Header space as "meta" information about the response and, since i view linking information is not "meta" but as "concrete", i place the links in the body of the response.

For me, the only exception is when the body type does not support links. For example, if it was important to add linking information to an image/* response body, I'd put those links in the LINK header field. FWIW, this rarely comes up in the work I do.

Hope this gives you some ideas.

mca

+1.859.757.1449
skype: mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://www.linkedin.com/in/mikeamundsen

-Ryan

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group, send email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft?hl=en.
 
 

[Glenn Block]

+1 to whatever Mike said. 

I haven't had to use this yet, but one great example I've seen is for geolocation. Imagine a Web API for mapping. I sent an accept header of "image/png" and it sends back a map. I then look to the links it returned in link headers which offer me options like move left, move right, move up, move down, zoom in, zoom out, etc.

The upside of links in headers is they work across any format. The downside is because they are separate, it's more work to parse, and because the links are not part of the payload, I might have to copy/persist both the links and the body  if I need to store the data for future.

[Andrei Neculau]

I have a bit more uniform view on this than Mike & Glenn, regardless whether the media-type is hypermedia aware.

Whatever items are part of the 1st level element "links", are also part of the Link header.

For now, my resources do not have contextual links (i.e. links within nested elements) - why not is part of another discussion. That may be an important factor, albeit you can still have something like:

Link: http://...; rel=...; context=/element/subelement

I personally like the Link header because it leverages a bit the power of hypermedia.

Given a house -> garage -> car -> owner pattern, if you're interested in getting a list of all the car-owners of a house, you could do just (I'm smashing URI opacity just for the sake of readability)

HEAD /house/1

 HEAD /house/1/garage

  HEAD /car/123

   GET /person/Andrei

  HEAD /car/456

   GET /person/Andrea

Now imagine this with no HEADs, but GETs [because the links are part of the payload alone], while the API client is not interested in any of those payloads.

Now, of course, if you detect this usage pattern of the API client, you might as well cater for this intent by providing "quicker" links,

but I think the beauty of a REST API is that it opens doors that the API designer never even imagined.

My 2 penny along with my reasoning, but far from a "crafter" :)

Good luck, Ryan!

[Ryan Connolly]

Mike, Glenn, Andrei: 

Thanks so much for taking the time to share your thoughts on this.  It seems even this thread proves my initial findings on there not being a real standard way of achieving HATEOAS and how tough it can be to achieve a good ReST architecture.  I think you all make valid points and your replies are very helpful indeed.  

Andrei: Are you saying that you include Link headers for GET responses as well as include them in the response body?  IOW, do you include Link headers only for HEAD requests or do you include them with GET requests for which you also return an entity that contains first level element "links"?  I'm guessing you are suggesting that Link Headers be used only in HEAD requests while GETs include "links" in the entity body (except for media-types you cannot possibly add them to like image*).

I guess for ultimate flexibility it would make sense to provide links in the body where possible and in the header, otherwise just in the header.  Does this make sense?  Any drawbacks to this approach?

Thanks again,

-Ryan

[Brian Mulloy]

i love this line:

It seems even this thread proves my initial findings on there not being a real standard way of achieving HATEOAS

everyday, everywhere on earth hundreds of millions of people use HATEOAS (Richardson Maturity Model Level 3) to surf the UI web with a standard browser. No problem.

yet about once a week we try to shoehorn HATEOAS into the API web.

Fielding's REST model was created by a PhD student observing the UI/brower web at the end of the 90s. It had nothing to do with web APIs.

It's been a decade. i just can't help feeling that HATEOAS is distracting us from creating other more interesting intellectual frameworks for practitioners building the app economy in 2012.

Let's cut the umbilical cord already and find a new abstraction to lead our efforts.

Who's got an alternative model?

[Peter Monks]

CORBA!!

(sorry, couldn't resist ;-)

Apologes for speling & gramar erorrs - sent from mobil deivce

[Ryan Connolly]

Fair enough, and I'm hearing your frustration. Clearly I do not have an alternative model in the works.

[Kevin Swiber]

On Oct 18, 2012, at 9:59 PM, Brian Mulloy <br...@apigee.com> wrote:

> i love this line:
>
>> It seems even this thread proves my initial findings on there not being a real standard way of achieving HATEOAS
>
> everyday, everywhere on earth hundreds of millions of people use HATEOAS (Richardson Maturity Model Level 3) to surf the UI web with a standard browser. No problem.
>
> yet about once a week we try to shoehorn HATEOAS into the API web.

You're being conservative. :)

> Fielding's REST model was created by a PhD student observing the UI/brower web at the end of the 90s. It had nothing to do with web APIs.
>
> It's been a decade. i just can't help feeling that HATEOAS is distracting us from creating other more interesting intellectual frameworks for practitioners building the app economy in 2012.

I wouldn't go that far. There is plenty of value in examining different architectures of distributed systems. The World Wide Web is a brilliant success (evidenced by how you end up at the weird part of YouTube when you search Google for "horse socks"). There's plenty to learn here.

> Let's cut the umbilical cord already and find a new abstraction to lead our efforts.
>
> Who's got an alternative model?

Hey, forget about strict client-server. Let's do distributed pub-sub in peer-to-peer networks of master-master systems. Nodes can subscribe to each other and are individually responsible for smart conflict resolution. Wait… I feel like I just described an XMPP/CouchDB mashup… over Erlang.

Maybe starting with a list of goals is better than starting with an existing spec, thesis, or piece of technology.


Regards,

Kevin

[Dan Schlossberg]

Good question, Brian. 

Whenever this topic comes up, it hurts my head.  It just doesn't seem to fit what I want to do with web Apis. I'm not building the next browser metaphor.  I want get the weather in LA or post on a friends wall or vote or send an an invoice or update my journal. 

So what would be a better model?

Lets take a look at what we are dealing with.  We've got some verbs and some state.  Now where have I seen those two suspects together before?  

Oh, No...

Oh, yes. 

OO

But wait, OO is too broad.  It seems we are changing state locally and persisting remotely.  (Get, update, delete) How about active record?  

Another way to think about it is odbc for the web. 

Don't like those models?  Your gonna hate these.  com... Rpc...  Corba...  

After all. A web API is an API. Its the web without the GUI.  http, tcp/ip is just transport.  

Hateos is dead. Hate something else. :)

Of course, I might be missing something.  I have thought about it much.  Like I said, it makes my head hurt. 

Dan Schlossberg 

Chief Software Craftsman

www.softwaresingularity.com

(m) 678.361.6960

Sent from my iPhone.

On Oct 18, 2012, at 9:59 PM, Brian Mulloy <br...@apigee.com> wrote:

[mca]

Brian:

you want to keep using the same protocol (HTTP), but use a diff way to map problem domains to it (i.e. not OO-CRUD not SOAP-TUNNELING over POST, not REST-based hypermedia messages)?

or do you want to use a new protocol (XMPP, WebSockets, etc.)?

maybe take the WebDAV approach and invent a new set of methods for HTTP that match closer to the general problem domains?

mca

+1.859.757.1449
skype: mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://www.linkedin.com/in/mikeamundsen

[Steven WIllmott]

I agree that it's worth searching for another model with some of (a lot of) the energy that goes into HATEOAS debates! Great topic :)

The biggest problem with HATEOAS model in my view is that, although it's incredibly flexible, it's reliant on having code which can on the fly interpret everything that's going on, to take advantage client code needs to be able to:

 - follow arbitrary links

 - interpreting media types

 - understanding the semantics of the links & objects on the fly

 - deal with change

Humans are awesome at this - their brains feed off all the cues they see and navigate the information at hand. They get to the semantics of what the intent of the API designer is (and the system underneath it). [Note, I'm not talking about the semantics of "GET/DELETE/PUT" - which is clear, but the semantics of the Data (e.g. Hotel, Car, Booking, Payment, ...).

So if you could build a client to do this on the fly, the model would be amazing. 

In reality you have to be able to write a robust bit of code which navigates an API or set of them with predictable results. 

I think it's agreed an API which is just a list of method calls is (the opposite end of the spectrum) very limiting - breaks every-time you change something. So my mind the search for an alternative framework should be somewhere in the middle of "hardcoded calls" and HATEOAS such that: 

 - An API can change dynamically within a certain envelope

 - It can be described/defined by a set of principles that don't have to explicitly enumerate

   every call, but in combination describe a set of API calls that can be made. 

 - Are tied to Mediatypes/DataModels 

So the API might have a more flexible structure, but there are certain things you as a client developer can depend on (a certain set of base media types, certain set of patterns of calls / URL structures). So moving from "I can throw anything I want at you at runtime" to "here are bunch of things that if you code against, you won't get a total curveball at runtime". 

I hate to give this as an example, but CSS is somewhat like this: it's a language for defining what something should look like with rules for the primitives and how they interact. This means to describe what you want you combine primitives, you don't have to enumerate everything explicitly. 

Allowing an API to throw different things are a client at runtime is extremely challenging to deal with. In any "new model" an intermediate step should be such that I can at least predict the space of those changes to an extent.

Second, the above refers to the API design framework and that's often the center of debate, but I think there is something even more important that would move the industry forward - higher level shared media types (but maybe data models is a better word): lots of them. Most media types now are things like JSON, Image, CSS, ... and have nothing to do with application level semantics. 

What would be much more powerful at shared concepts of things like: 

 - hotel 

 - car

 - rental 

 - house

 - user

 - currency

 - cameras

 - ...

This is because as a client developer, the idea that a hotel is represented in the same way on all the travel sites, all the travel APIs etc. you might use (or least in the same basic way but maybe with some extensions) is a fantastically large time saver. It allows you to build a much richer, more robust client. You can get on with pulling "hotels" out of these APIs, no "JSON". About the only common media type today that gets close to this is vCard.

There are plenty of semantic web efforts around these types of "higher level" definitions and it can be a hornet's nest to define them. There are also efforts in areas like travel to define data model standards (e.g. OTA). But, for APIs to scale out to many 100's of thousands, I think convergence on this level is incredibly important. Probably more important than what verbs etc. are being used and how. Also, to be clear I'm not really that concerned about semantic web type hierarchies of concepts, inferences of one from the other etc. (although that would be nice), but plain and simple "this API, uses THIS data model" and oh, it's that's the same one I programmed against last week somewhere else. 

The reason HTML+Browsers work so well in HATEOAS mode is: they rely on the human brain to do the ultimate interpretation and navigating of what's being rendered. What APIs are often trying to do is get by without a human in the loop. 

Unfortunately our ability to build such smart clients is limited, so any framework needs to be structured enough to allow code to be written.

I'm not down on HATEOAS - it's actually (I think) ultimately close to what we want + I'm certain we can do better than what is done today (long lists of example URLs), but it has to remain in the reals of the doable for client programming.

  steve.

[Dan Schlossberg]

Umm. Dont You want to use the http protocol because it already maps to the crud domain so well. 

No need to invent anything new. This fielding character already took care of that.   He was smart enough to see that we really don't want to do anything new with the web. Crud works for everything. 

Btw. That's two votes for corba.  Do we have third?  

Dan Schlossberg 

Chief Software Craftsman

www.softwaresingularity.com

(m) 678.361.6960

Sent from my iPhone.

[Mike Schinkel]

On Oct 18, 2012, at 11:05 PM, Steven WIllmott <ste...@gmail.com> wrote:

The reason HTML+Browsers work so well in HATEOAS mode is: they rely on the human brain to do the ultimate interpretation and navigating of what's being rendered. What APIs are often trying to do is get by without a human in the loop. 

That right there is the key insight.

I propose the new model be "Amazon's Mechanical Turk, the REST edition."  Yoru API client sends ambiguous requests and a human on the other end figures out what you really wanted and return you exactly that.  

Of course I haven't worked out a solution to the latency problem....

-Mike

[Steven WIllmott]

On Oct 18, 2012, at 8:15 PM, Mike Schinkel wrote:

On Oct 18, 2012, at 11:05 PM, Steven WIllmott <ste...@gmail.com> wrote:

The reason HTML+Browsers work so well in HATEOAS mode is: they rely on the human brain to do the ultimate interpretation and navigating of what's being rendered. What APIs are often trying to do is get by without a human in the loop. 

That right there is the key insight.

I propose the new model be "Amazon's Mechanical Turk, the REST edition."  Yoru API client sends ambiguous requests and a human on the other end figures out what you really wanted and return you exactly that.  

Damn that's a genius solution!

Of course I haven't worked out a solution to the latency problem...

your client development time is near zero though!

-Mike

[Glenn Block]

HATEOAS doesn't require you to be RESTful to use it. I am not sure what him being a PHD student has anything to do with it. That same student just happened to design HTTP itself as well much early web infrastructure.

Anyway, i think your argument is pretty weak. I do agree that there's way too much emphasis on REST.....

Linking can be used i many different types of systems as a way to decouple clients and servers.

[Mike Kelly]

On Fri, Oct 19, 2012 at 2:59 AM, Brian Mulloy <br...@apigee.com> wrote:
> i love this line:
>
> It seems even this thread proves my initial findings on there not being a
> real standard way of achieving HATEOAS
>

We're still in the very early days of applying hypertext to APIs, so
it's natural that we haven't achieved a 'real standard' yet.

> everyday, everywhere on earth hundreds of millions of people use HATEOAS
> (Richardson Maturity Model Level 3) to surf the UI web with a standard
> browser. No problem.
>
> yet about once a week we try to shoehorn HATEOAS into the API web.

I'm not sure what you are getting at here, please could you expand?

> Fielding's REST model was created by a PhD student observing the UI/brower
> web at the end of the 90s. It had nothing to do with web APIs.

don't agree because it was higher level than that

"REST provides a set of architectural constraints that, when applied
as a whole, emphasizes scalability of component interactions,
generality of interfaces, independent deployment of components, and
intermediary components to reduce interaction latency, enforce
security, and encapsulate legacy systems."

> It's been a decade. i just can't help feeling that HATEOAS is distracting us
> from creating other more interesting intellectual frameworks for
> practitioners building the app economy in 2012.
>
> Let's cut the umbilical cord already and find a new abstraction to lead our
> efforts.
>
> Who's got an alternative model?

Most "REST" APIs are actually using an alternative model to REST. What
apiary are trying to do can be looked at as an alternative to REST.

Cheers,
M

[Andrei Neculau]

Back to the topic at hand 

On Friday, October 19, 2012 3:19:15 AM UTC+2, Ryan Connolly wrote:

Andrei: Are you saying that you include Link headers for GET responses as well as include them in the response body?  IOW, do you include Link headers only for HEAD requests or do you include them with GET requests for which you also return an entity that contains first level element "links"?  I'm guessing you are suggesting that Link Headers be used only in HEAD requests while GETs include "links" in the entity body (except for media-types you cannot possibly add them to like image*).

Despite the duplication, both (HEAD and GET response headers should not differ). I have the links in the body just for visibility, and also because it is not meta-data (as Mike pointed out). I personally don't take this as a DRY issue - same thing happens with the self link.

[Andrei Neculau]

Most "REST" APIs are actually using an alternative model to REST. What
apiary are trying to do can be looked at as an alternative to REST.

Cheers,
M

Care to explain the bit about apiary? Where's the alternative?

Although I'm in love with apiary.io and the guys behind it, all I see is top-notch leveraging of L1+2 REST mocking, docs, etc, otherwise it's purely a HTTP API tool.

Thanks

[Ryan Connolly]

On Friday, October 19, 2012 6:28:33 AM UTC-4, Andrei Neculau wrote:

Back to the topic at hand 

On Friday, October 19, 2012 3:19:15 AM UTC+2, Ryan Connolly wrote:

Andrei: Are you saying that you include Link headers for GET responses as well as include them in the response body?  IOW, do you include Link headers only for HEAD requests or do you include them with GET requests for which you also return an entity that contains first level element "links"?  I'm guessing you are suggesting that Link Headers be used only in HEAD requests while GETs include "links" in the entity body (except for media-types you cannot possibly add them to like image*).

Despite the duplication, both (HEAD and GET response headers should not differ).

Right.  Thanks for the reminder on this.

 

I have the links in the body just for visibility, and also because it is not meta-data (as Mike pointed out). I personally don't take this as a DRY issue - same thing happens with the self link.

Now if I may, I would like to open up that other discussion you mentioned regarding contextual links.  What is your reason for not including these at this time?  

[Mike Kelly]

There aren't layers of REST. There's a set of constraints, only one of
which is optional (code on demand). If you only meet a subset of those
required constraints then what you are doing is not REST, by
definition. I know saying that upsets people because it messes with
their marketing, but it's written in black and white in the
dissertation. The style apiary are supporting is not REST.

To be clear I'm not saying that in a demeaning way, there's not
anything inherently "wrong" with doing something other than REST.

I was making the point to Brian that there are plenty of examples of
alternative models to REST in the wild and that (confusingly) these
are exhibited by the very APIs that claim to be "REST" but actually
aren't.

Does that clear that up or did I make it worse!?

Cheers,
M

[Kevin Swiber]

On Oct 19, 2012, at 7:44 AM, Mike Kelly <mikeke...@gmail.com> wrote:

I was making the point to Brian that there are plenty of examples of
alternative models to REST in the wild and that (confusingly) these
are exhibited by the very APIs that claim to be "REST" but actually
aren't.

I know. I'm with you. It's weird. I often say the word REST has taken on a different meaning than its original intent. But we lost the battle. Majority rules, and we're not in it. Neither is Fielding. So it goes. 

[Andrei Neculau]

By all means Mike, you made my day :)

I am with you 100% - there is no less REST or more REST imho, but for the sake of moving on with more concrete topics, beyond "this is (not) REST", I am often giving in the perspective of "RMM layered REST".

"The style apiary are supporting is not REST." Agreed, and I believe Jakub is also aware of it, but as Kevin wrote.. most often REST means smth else than what this group calls REST, and I think that's why you see "REST API documentation" on apiary.io

Despite all the level that we agree on, the question still remains though. I don't see Apiary providing an alternative to REST. It covers the whole world of HTTP APIs. But that's just me.

[Andrei Neculau]

Pandora's box...

Disclaimer: Take a big breath before the following. Most of it is not properly digested, just.. some food for thought.

If I remember correctly Matt Bishop is making use of contextual links (see https://vimeo.com/49609648 )

BUT he also makes a nice highlight - his resources carry very little information, mostly links to other resources, and also attach a self-link to representations of related-resources in-the-making. Albeit, you don't always get what you want, I would totally strive for this. It's a pure perspective - don't you just like this "the item doesn't have a price, it's just the price that knows about the item" ?

The only exception I would make is for a zoom-like functionality that Matt talks about. Although, the zoom functionality works more as a response packaging with a root reference ( Give me /x, and while you're at it, also give me link x.y and link x.y.z ). That said, you still don't have contextual links per se - it's just that the response is a representation of multiple resources. Do you follow me?

What Matt is achieves (my POV) is exactly an alternative towards leveraging hypermedia, but also incorporating some chattiness aid, which beautifully addresses what I wrote in the first reply:

Now, of course, if you detect this usage pattern of the API client, you might as well cater for this intent by providing "quicker" links,
but I think the beauty of a REST API is that it opens doors that the API designer never even imagined.

Although there are more experienced "crafters" on this group, and I can guess that I'm in minority with this vision, I keep to its simplicity until I see what opportunity I am missing on: it is resources that have links, not the contextual bits.

And yes, I carry the same vision regarding HTML for instance. If you collect all the LINK tags, and move them to the bottom of the BODy, this still holds "the resource at HREF is in a RELation to the current resource". The context is only for displaying/human purposes (links are displayed somewhere else and also may not make sense for the human looking at them), and so far I am yet to see an API that has presentation-aware messages. IMHO, presentation should not be mixed with the message, albeit the message can have presentational data (i.e. a template property {name: "Andrei", greeting: "Hello!", tpl: "<b>{name}</b>: says {greeting}"}, or a CSS link, etc)

[Steve Klabnik]

> It's been a decade. i just can't help feeling that HATEOAS is distracting us
> from creating other more interesting intellectual frameworks for
> practitioners building the app economy in 2012.
>
> Let's cut the umbilical cord already and find a new abstraction to lead our
> efforts.

I had five startups this week tell me that they're building
fully-HATEOAS APIs, and that they'll never go back. GEICO is known to
be working on a HAL API, and I know of two other Big Name 'startups'
that are too.

It's still bleeding edge. Don't get so down so quickly. ;)