Help me, I'm having a RESTistential crisis

[Evan Cordell]

There are several things that have been bothering me about (web) APIs, REST, Hypermedia, SOAP, and so on. I'm sure most of this comes down to something that's just not quite clicking, so I invite you to correct my misconceptions and explain why my concerns are unfounded. I'll try to be as coherent as possible as I attempt to structure the stream-of-consciousness notes I've collected on the subject.

I like to think I've done my homework. I've read Fielding's dissertation, the majority of a few books (some of which were written by people who weigh in on this group from time to time, which is laudable), and countless blog posts/articles. But I'm definitely not an expert in the field, and this seems like a great place to get feedback on my RESTistential crisis.

REST (hypermedia in particular) seems to be fundamentally at odds with API Design

Conventional API wisdom is that a good API doesn't change. This isn't often stated explicitly when talking about a "traditional" API (though Joshua Bloch mentions the immutability of Public APIs in his excellent talk), but nevertheless we don't expect the NSString class to change its interface during execution (or if it's going to do so, we expect it because of other local code). Changes are not introduced into a local API by allowing a server to remotely update your local libraries at will - instead, libraries are versioned, and generally a developer only updates those libraries for non-whimsical reasons (features, deprecation, platform support, etc). And the more common an API the less we expect to deal with changes to it - you wouldn't bother wrapping the Java String class in an adapter, but you probably would want to wrap YourFavoriteJSONParser, so that you could easily switch out the implementations later and/or isolate breaking changes between versions.

This sort of strong interface guarantee doesn't exist in a web API, but it's exactly the sort of guarantee one wants, as an API Designer, to be able to make. Which sort of begs the question: how does the REST architecture deal with a changing interface? I think most people (and this is parroted pretty often on this group) would suggest versioning the API as well.

Versioning Doesn't Work

It seems to me that there are some inherent problems in versioning a web API. Sure, versioning is a great idea when there are small changes in a representation. You can continue to serve two representations (versions) of the same resource with very little problem.

But what about when you change the underlying resources? What about when your problem domain evolves as it inevitably will and you need to change with it? Do you run two separate stacks? For how long? How many clients will you intentionally break and force to update to the new version?

More fundamentally: a "Web API" exposes far more than a "traditional" API. It exposes data, and more often than not it exposes the current state of a large set of constantly-changing data. How can you effectively version when your domain model has two versions, which share some subset of data, but are accessed in different ways? You force the clients to break.

But, of course, that's not really worse than a non-RESTful web API. Clients would break anyway, and at least with hypermedia you have the ability to make small changes to the domain model without breaking clients. Certainly that's worth it?

Hypermedia Destroys Resource Location

Except that the way in which hypermedia accounts for those changes breaks one of the other useful things about a web API: the URI.

Fielding has a few things to say about this on his blog: 

A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC's functional coupling].

 

A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]

I'm pretty sure I understand the impetus for this: following this pattern, you can never make a "bad" request to a server (as he states, it removes coupling between client and server). On the other hand, users never follow this. Browsers have bookmarks, and people use them, which is why redirection has become such an important thing to keep in mind. Similarly, an API client can never make any assumptions about the resources that lie at any given URI - unless they have started at the root URI and navigated there. 

The arguments for hypermedia make sense for human users, not machines, and they don't deal well with change over time.

This approach makes a ton of sense when the viewer of the REpresentation is a human. Humans can read descriptions and decide which action to take, which state transformation to initiate, which link to follow. Machines on the other hand - not so much. When you provide an API client to an application developer you want to be able to guarantee that when they call library.search('Kristmas Karol') it actually initiates a search on the server and returns the result. But following a hypermedia design, what you really need to do is ask the server to list the things it has, see if a library is one of those (and hope it hasn't changed its name), and hope that search is an action you can perform on it (and hope it hasn't changed its name). This is inherently difficult to automate. 

A purported advantage of hypermedia APIs is that they reduce client breakage and increase API longevity. But an API client is useless if not consumed, and changes in the domain model necessitate changes in the way it's consumed. So does hypermedia really buy you anything? 

Does an API really need to be inherently discoverable? What's so bad about out-of-band?

It seems clear to me that the ideas behind e.g. HAL are good for the reason that developers can discover the API. "Normal" users will not use these APIs in a way that discoverability is useful. So the question becomes: is it necessary that an API's discoverability be self-documented, or should it just have "discoverable" documentation? In other words: why should the api be self documenting, when the format of the API responses are difficult to read at high levels of complexity. You know what's not difficult to read? Nicely structured HTML, rendered by a browser, with interactive examples and articulate descriptions.

Is a hypermedia API really that different from a WADL?

I've read some pretty good arguments against WADL. But when it comes right down to it, what's so different about a discoverable, hypermedia API? A "HAPI" just seems like an API with the WADL distributed throughout. You still have to start with a single endpoint, and you still have to figure out what you can with the API based on that response.

The article that I linked suggests, as Fielding does, that media-types are the way to go. But to me, that just seems like moving the description elsewhere (and only very slightly). Something has to have a description of the domain models, and that something will change over time. Am I wrong about this?

A Web Analogue

Consider the common scenario of using a REST API in a mobile application. Suppose you had two languages: Android ML and iOSML, that could completely describe an Android or iOS app, respectively (and let's assume Code-on-demand while we're at it). 

It seems like that type of scenario is the only way you get to have all of the nice properties of the Web (i.e. a REST architecture) by using other media types. If you break down your domain objects into their own media types, you still have to write custom software that understands it that is specific to your API. It seems to me that the only way to get truly reusable media-types is to make them sort of platform specific as I described. Then you can reuse your media type to describe any problem domain, and transfer the appropriate representation to any device of the same type. 

What seems to really be good about REST

More so than hypermedia, it seems like the things that are really amazing about a REST architecture are:

- Enforcing a stateless server makes it very easy to scale

- Thinking in terms of resources encourages decoupling of information and simple interfaces

- The basics are so simple that you can talk to servers in a semantically meaningful way with nothing but the http library included on every platform and language

I'm not as angry as I may seem

The history of this tirade is that I've been becoming more and more convinced that hypermedia is the holy grail: it seems like, once you have hypermedia, you can start to do less work on the client. When you have a general client for a standard hypermedia format, you have to do even less client work. It seems so perfect. And the more I think about it, the more it seems like you could do a lot of code generation. Couldn't you write a document (Apiary has some good ideas here) that describes your resources and state transitions, and be done? A sufficiently descriptive document of that type, with proper tooling, could be used to generate a web server, and clients in all sorts of languages. The steps between thought and product are dramatically reduced.

But that line of thinking threw up some warning flags for me, and I started doing a lot of research on SOAP and WSDL and WADL (things which I admit I've had little to no experience with), and I came to fear my idealization of these concepts. This document is a transcription of the frustration I feel at being unable to adequately form an opinion on the subject. 

So please, help me API Craft. You're my only hope.

[Richard Mateosian]

Josh Bloch is also the author of Effective Java, which I had the pleasure of reviewing in IEEE Micro in 2002 (http://xrmcontent.blogspot.com/2002/08/effective-java-c-essentials-essential.html).

Josh has a rare combination of deep detailed knowledge and the ability to think simultaneously at several levels of abstraction. He is also (or perhaps consequently) a great explainer.  If you ever get a chance to hear him talk, don't miss it.   ...RM

On Wed, Feb 26, 2014 at 3:49 PM, Evan Cordell <cordel...@gmail.com> wrote:

Joshua Bloch mentions the immutability of Public APIs in his excellent talk),

--

Richard Mateosian <x...@pacbell.net>
Berkeley, California

[Jerome Louvel]

Hi Evan,

I think you've done a pretty accurate analysis of the situation. We need more clarification of the difference and complementarity between the formal REST style and the informal Web API style.

REST has been designed on purpose with hypermedia and human users in mind. Even though its principles are very valuable to machine driven web API, there is a need for a separate architecture style.

I have started an effort to formalize this Web API style, contributions welcome:

https://github.com/restlet/web-api-style

Jérôme

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at http://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/groups/opt_out.

[mca]

Jerome:

Love that you reference Taylor's book, too. Great stuff in there.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[mca]

I lean on Joshua Bloch's "Characteristics of a Good API"[1] quite often when designing programming interfaces for the web. I esp. like referring to both Bloch's list and Fielding's "Rules for Hypermedia-Driven APIs" since I think they compliment each other very nicely.

[1] http://lcsd05.cs.tamu.edu/slides/keynote.pdf

[2] http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

--

[Jerome Louvel]

Mike, this book is great indeed and gives a deeper view on REST, beyond Roy's dissertation.

I also like the part of the architecture definition languages.

Jerome

[Jerome Louvel]

Thanks for sharing Mike. Must read resources indeed.

Just updated the references page to mention them:

https://github.com/restlet/web-api-style/wiki/References

Jérôme

[mca]

Yes - Taylor's primary focus is on ways to create there architectural "languages" - some very intriguing things there.

And i like that the Taylor text actually goes will beyond REST into other styles that work in more integrated stacks like the Phillips case.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Jørn Wildt]

Evan, I've been through the same line of thoughts and, yes, REST is different. But hyper media and explorability makes a lot of sense if you dig into it - also for machine-to-machine communications. I have tried to explain it in a series of blog posts.

First we have hypermedia; there was a great discussion on this list for some time ago "Selling the benefits of hypermedia". I have wrapped it up in this blog post: http://soabits.blogspot.com/2013/12/selling-benefits-of-hypermedia.html - the main "take away" is that hypermedia has a few benefits for one-off/snowflake/single-instance API implementations but it is actually a necessity when you start looking at services defined without reference to any specific implementation.

Then we have the citations you mention from Fielding's famous online rant. I have described one valid way of understanding it and describing how it could be applied to something like an issue tracker here: http://soabits.blogspot.com/2013/05/the-role-of-media-types-in-restful-web.html. The main take away here is that it is possible to describe a "API" without referring to any specific implementation of it. As a follow up later on I wrote this one: http://soabits.blogspot.dk/2013/12/media-types-for-apis.html

As I side note I have also touched upon error handling in a REST setting: http://soabits.blogspot.dk/2013/05/error-handling-considerations-and-best.html (mostly the last section).

Lately I have taken all that stuff and wrapped it up in a new media type called Mason: http://soabits.blogspot.dk/2014/02/implementing-hypermedia-apis-and-rest.html. Then I have used Mason to demonstrate how a REST service can be described (http://soabits.blogspot.dk/2014/02/representing-issue-tracker-with-mason.html) and shown an concrete implementation of that service (http://mason-issue-tracker.cbrain.net/resource-common). It is my hope that such a concrete manifestation of a suitable media type and a example REST service can show how REST works - also for APIs and M2M scenarios.

I know thats a long read, but its also a complex question you give :-)

Regards, Jørn

[Mike Kelly]

You seem to have a pretty solid understanding.

First off, it's more than possible that you're facing problems in
which the full-blown REST style is not applicable.

Having said that, it's also important to know that there are different
approaches to hypermedia APIs; some are more complicated than others
(i.e. introducing dynamic affordances like 'forms'). If you're feeling
like the more complicated hypermedia approaches are "not clicking"
then you're in the same boat as many others (myself included.. and
I've been trying for a while). Simpler approaches to hypermedia that
avoid forms and stick to discoverable but human-readable documentation
are not necessarily 'out-of-band' and still constitute
REST/hypermedia.

In other words, if you feel like all this REST/hypermedia stuff is "a
bit too much" it might just be that you are getting some bad advice on
the topic. Or not..! It's difficult to say without understanding your
wider context (i.e. what kind of problems you are solving).

Good luck.

Cheers,
M

[Evan Cordell]

Jerome:

This is an incredibly useful resource - I knew people must be discussing these sorts of things, but Googling for REST related topics tends to dig up only a particular type of article.

The "Web API Style" set in comparison with REST style articulates a lot of what I was trying to express above very well (although there are still some gaps that it would be interesting to think about filling).

The main hangup I'm having is that some of what REST (and I would say particularly re:hypermedia) guarantees by design is not generally what you want an API to guarantee by design. Your description of a "Custom Interface" brings a lot of that into sharp focus.

[mca]

<snip>

The main hangup I'm having is that some of what REST (and I would say particularly re:hypermedia) guarantees by design is not generally what you want an API to guarantee by design. Your description of a "Custom Interface" brings a lot of that into sharp focus.

</snip>

I'd love to hear more about this.

What are the "guarantees" you  are talking about? and why are these things "not generally what you want"?

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Evan Cordell]

I'm currently working through all of the great information linked in this thread, so forgive me if this is addressed somewhere above.

I'm basically asserting that:

1. An important, maybe the most important, feature of a good API is that it does not change (especially while you are using it).

2. REST/Hypermedia intentionally address the problems of a changing interface

And that therefore they are in some ways "spiritually at odds" with one another.  

[mca]

ok i think i see your POV.

you like #1, right? you like that notion that change won't happen. and you think #2 is basically at odds with that. #2 claims that change *will* happen you can use hypermedia to mitigate the changes.

do i have that right?

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Peter Hamilton]

Hypermedia defines How (method, data) and Where (URI). I think you are right about changing the How, but I think one of the beauties of hypermedia is being able to change the Where without breaking the API.

[Pedro Santos]

It all depends on how you look at "change" within the context of a hypermedia-capable API.

Warning: Weird convoluted ideas may follow :-P

I think Josh Bloch's argument about unchanging API's is about unchanging expectations. If as an axiom you accept that the capability of change is *part* of an API design, then suddenly all "change" forseeen by the API design is not really change per-se (i.e. a violation of expectations put forth by the API), it's just the API being what it is (i.e. effectively staying the same).

For example, consider the case where you design an "API" that's just exposing a specific implementation. Any client using that API knows at start that once and implementation changes the API will also change. Your API has zero flexibility. The expectation is that the API will always be the same and that implies that in order to change the implementation you will have to change the API.

Now consider an API that follows an interface/implementation design. That separation, allows for some change (of the implementation) and yet the expectations about the API (i.e. it's interface) doesn't change. Now you have a little flexibility, change is actually possible without changing the API.

In REST, you can actually go a step further. Your API design explicitly allows for change. You can argue that changing URIs (for example) can happen as part of you API specification, and provide for both servers and clients mechanisms to deal with that (in this case changing URIs are *part* of the API, just like the ability to change an implementation is part of an API that follows the interface/implementation design).

So in short, an hypermedia API is always the "same", i.e. always having the ability to change (to some degree), and eventually doing so.

I don't know if I'm making sense... it does to me :-)

Cheers,
Pedro

[Mike Kelly]

On Thu, Feb 27, 2014 at 11:13 AM, Evan Cordell <cordel...@gmail.com> wrote:
> I'm currently working through all of the great information linked in this
> thread, so forgive me if this is addressed somewhere above.
>
> I'm basically asserting that:
>
> 1. An important, maybe the most important, feature of a good API is that it
> does not change (especially while you are using it).
> 2. REST/Hypermedia intentionally address the problems of a changing
> interface
>
> And that therefore they are in some ways "spiritually at odds" with one
> another.
>

A good API won't make breaking changes. The challenge is finding
strategies to make sure you don't do that. Hypermedia can help.
However, if the design is too complex and people can work around it
then they will - and you'll be left with a false sense of security in
which a change you've designed to be non-breaking actual does cause
breakage in practice because you can't (and shouldn't) control all the
clients to your API.

Cheers,
M

[Evan Cordell]

Mike, I think you understand my position. I 'like' the notion that change won't happen (but more than that, I'm asserting that that's an important part of a "good" API). 

A REST architecture addresses a lot of problems really well, but it explicitly rules out the notion of an unchanging interface in favor of hypermedia.

Peter responded as I was writing: "one of the beauties of hypermedia is being able to change the Where without breaking the API"

I do see the value in that. But if all you're changing is the Where, maybe you should just not be changing the Where (i.e. hypermedia doesn't let you address any "meaningful" changes.) It's almost like hypermedia lets you address "patch-level" versioning, but it doesn't seem to handle "minor" or "major" versioning. 

Suppose you have a small collection of Books, and your Book resource has an "open", "read", and "close" action, and your client is designed to let a user look at all of the books, open a book, read it, and then close it. Now suppose you decide to include the notions of searching for Books in a Library, and that you must "check out" a Book from a Library in order to open, read, or close it. The only way to deal with this change is to a) rewrite the client to use these new notions or b) have the interface to the resources be directly generated from the API responses - this is the case of HTML/Web Browsers, but it's just not how APIs are consumed.  

I'm trying to wade through all of the links Jørn gave, which I think may address some of this.

[Pedro Santos]

Suppose you have a small collection of Books, and your Book resource has an "open", "read", and "close" action, and your client is designed to let a user look at all of the books, open a book, read it, and then close it. Now suppose you decide to include the notions of searching for Books in a Library, and that you must "check out" a Book from a Library in order to open, read, or close it. The only way to deal with this change is to a) rewrite the client to use these new notions or b) have the interface to the resources be directly generated from the API responses - this is the case of HTML/Web Browsers, but it's just not how APIs are consumed. 

This is an application-level change. You basically introducing new application-level semantic (the concept of "checking out" a book) and also changing the application workflow (i.e. must "check out" a book before being able to do the previous actions of "reading", "opening" and "closing" the book).

Although hypermedia can be helpful it's not magic. The introduction of application-level semantics that afect the application workflow requires either an intelligent client (as in a human behind a browser, in the future who knows...) or a client that already understands the new semantics (if for example the media type already had the concept of "checking out", but this particular API didn't use it before). In both these cases the client is able to function as normal because of the existance hypermedia controls.

If however the client doesn't understand the concept of "checking out" and has no capability to learn new concepts, then no amount of design can help it. The new concept and workflow change is effectively changing the clients expectations and thus introducing a breaking API change (for clients that are not capable of learning new concepts or don't understand the new concept).

If an "automated" client, with limited understanding of a problem domain, in any way relies on that understanding to perform a number of functions correctly, and somewhere in time somebody alters the necessary understanding to perform those same functions then, to allow for this change, either:

a) The client is able to "learn", and given enough information can decide what to do, or
b) The client's understanding of the problem domain already included the introduced changes and it was already "prepared" for any configuration within it's understanding

Even inteligent machines (like humans) have a limit on what they can understand, and although we can handle a lot of change, even with the proper tools, some change can be "breaking" change for some (or even everyone).
Hypermedia is just a "tool" to make *some* previously breaking changes not breaking for those that are prepared for it. But it won't solve all conceivable braking changes in an API.

[Brad Jones]

Guys, I'm going to jump in here and be WAY over my head because I'm neither a programmer nor a data scientist: Does the JSON-LD/Hydra concepts not address the "How (method, data)" somewhat? see: http://www.markus-lanthaler.com/projects and http://www.slideshare.net/lanthaler/building-next-generation-web-ap-is-with-jsonld-and-hydra

-- Brad Jones

"dumbass business guy" who wants a declarative world for data

[Jørn Wildt]

Evan> But if all you're changing is the Where, maybe you should just not be changing the Where (i.e. hypermedia doesn't let you address any "meaningful" changes.)

Here is one absolutely valid reason for changing the "Where" in an API: off-loading resources to a Content Delivery Network (CDN). Assume you have your book resource including the ID of the image of the book cover:

{

  Title: "Rest in Practice",

  Authors: [ ... ],

  CoverImageId: 1234

}

At first your client is hard coded with the algorithm: "when I have a cover image ID I must append it to the URL http://books.com/covers/".

Later on the API off-loads all the cover images to a CDN . Now the cover image URL changes to something different like http://some-cdn/xys/blob-881j7d. Duh! All clients break at once - full stop.

Had the response contained hypermedia, links to the image covers, there would have been no breaking changes as the server response changes from

{
  Title: "Rest in Practice",
  Authors: [ ... ],
  CoverImageUrl: "http://books.com/covers/1234"
}

to


{
  Title: "Rest in Practice",
  Authors: [ ... ],
  CoverImageUrl: "http://some-cdn/xys/blob-881j7d"
}

More example here: http://soabits.blogspot.dk/2013/12/selling-benefits-of-hypermedia.html

/Jørn

[Evan Cordell]

Jørn, that is a lot to read and think about, thanks for all of the information.

In the first link, "Selling the Benefits of Hypermedia", I think I have a contention with the first assertion: "It may sound trivial but do not underestimate the power of an explorable API." I definitely agree that an explorable API is easier for a developer to grok and begin using. But the cost is that not only is the way you read it explorable, the way you use it is explorable as well. 

You could imagine a javascript wrapper around a hypermedia api that abstracts the interaction a little bit. Using your mason issue tracker example, consider a javascript call like: api(http://mason-issue-tracker.cbrain.net/resource-common).links.logo to get the logo for the service. Once you write this line of code, all bets are off. You've now automated the selection of the is:logo link. If you (the service provider) choose to, you could completely remove the logo from the original representation, and the client would break (you wouldn't do that in this scenario, of course, but the example is contrived). The point is simply that a consumer of a web API lacks, inherently, the contract that an API should really provide (and again, that's just an assertion I'm making). The guarantee comes down to a personal one, "I promise I won't make breaking changes, and I promise that I fully understand every possible way in which a change could be breaking, despite the monumentality of that task."

There's another aspect to this of course, which is simply that network communication is inherently faulty. There's no guarantee that a message you send from one node will make it to another, or that if it does you'll get a response. I think you just have to accept that, in a service oriented architecture, the service could be down. But is it also necessary to accept that the requests that you do send might fail because the service itself decided to change?

So I guess: is that the best we can do? How can an API consumer have confidence that the semantics of the API won't change? I've done some reading on Netflix's api and, if I understand correctly, clients basically get to define the type of response they want back. That definitely seems like a stronger sort of guarantee than hypermedia gives you. Or I could imagine some sort of system like Git, which manages to be distributed and asynchronous but which also guarantees that you're seeing the same exact content as everyone else who has replayed a certain series of commits (at first glance this has a lot of other problems when applied to a Web API, though).

Hypermedia gives you some degree of freedom in the changes you can make without breaking a client. But even with hypermedia, there is no strong guarantee that what you ask for is what you will get (which, again, I would emphasize is contrary to what an API should be). So I can't help but wonder if the complexity required to implement hypermedia in both client and server is "worth it."

It seems like a good set of rules might be:

- Version the API fairly granularly

- Have URIs that have semantic meaning and can be understood (I might go so far as to say constructed, but I haven't convinced myself of that)

- Have really good documentation.

- Have a way to ensure that your documentation is an accurate description of your API, for all versions

Anyway, back to the article:

3. Simple client logic
A client that simply follows URLs instead of constructing them itself, should be easier to implement and maintain. It won't need logic to figure out which values to substitute into what URL templates. All it has to do is to identify links in the payload and extract the hypermedia reference URL.

I don't know that I find this convincing. I definitely agree that pointing to content resources with a url makes sense, as you pointed out. But don't you need URI templates generally anyway? If you allow any sort of querying, you need to be able to construct a url based on some parameters.

The use of hypermedia removes the client's hard coded knowledge of the URL structures used by the server. This means the server is free to change its URL structures over time when the API evolves without any need to upgrade the clients. 

Yes...but now a client has no reason to believe that a resource that it got once could be retrieved from the same place (you can't cache urls). On the surface, makes URIs so incredibly fragile and ephemeral - you only really know you can use it if you use it and it works. In practice of course, this is avoided by using redirects and the like, but then...if you do that, you don't really need hypermedia affordances, right? Your urls can change over time, and with proper redirection, the client doesn't need to know anything about the hypermedia format you're using, just how to follow redirects. Then you can cache urls, and if they change, well, cache the new location that the redirect response tells you about.

I definitely do see the utility in referring to resources by their URI rather then some id or other. It does sort of seem like there are two "levels" of hypermedia integration: simply referring to locations of resources with a URI, and defining all possible actions on a resource by way of URIs. The latter seems fragile to me, because actions performed on a resource require the API consumer to make decisions. (In summary: definitely agree with your point about CDNs, but that's not really what I'm referring to by hypermedia. I was thinking more along the lines of the lists of "links" and perhaps even forms that hypermedia formats specify). (I'm not finding it right now, but I'm pretty sure Mike Amundsen has some writing on the subject of different types of hypermedia affordances - if I can find it, or you or he can point me to it, I'll define exactly what I'm talking about so we're using a common vocabulary).

I feel like I'm rambling now, and that's just a response to some of what was in one of those links. The link about versioning speaks to a lot of the concerns I have, but I don't feel like I've adequately digested all of it to make any coherent statements or ask any good questions. 

I'd be interested to know what you think about my theoretical proposal for platform-specific media types, which seems to me to fix some of these problems (basically, because it removes the hard-coded m2m part).

I did read the other links you posted, but there wasn't anything that I felt explicitly addressed my concerns (very good reading though, thanks). I really like the error handling work you've done (and I like Mark Nottingham's proposal as well). Mason seems to still have the same issues that I'm concerned about, and I'm also not overly clear on the similarities/differences with other hypermedia json media types. Oh, and your description of the BugTracker media type was very interesting, and I like the idea of non-domain-specific media types.

[Jørn Wildt]

> I definitely agree that an explorable API is easier for a developer to grok and begin using. But the cost is that not only is the way you read it explorable, the way you use it is explorable as well.

Why is that a cost? Why is "the way you use it explorable as well" an additional cost? Sorry I don't get that?

> Using your mason issue tracker example, consider a javascript call like: api(http://mason-issue-tracker.cbrain.net/resource-common).links.logo to get the logo for the service.

> You've now automated the selection of the

is:logo link.

Yes.

> If you (the service provider) choose to, you could completely remove the logo from the original representation, and the client would break (you wouldn't do that in this scenario, of course, but the example is contrived).

Correct, but as you say, why should the service provider do that? I'll give a less contrived example below.

> The point is simply that a consumer of a web API lacks, inherently, the contract that an API should really provide

Allow me to disagree with this. The contract is either in a domain specific media type or a generic one plus some sort of "profile" document (like my issue tracker description). That contract (the media type/profile) can/should specify which link relations a client can expect to find in the various responses. Some of these can be required - but the logo may not be, as not all issue trackers can be expected to have a logo, which means you coded your client wrong - it should test for the existence of the "logo" link first before assuming it can be used. My "contract" is similarly too loose - it should have stated for all link relations whether these are required or not.

> The guarantee comes down to a personal one, "I promise I won't make breaking changes, and I promise that I fully understand every possible way in which a change could be breaking, despite the monumentality of that task."

Yes. That is how you must think when evolving a public API where you have no control of the clients using it.

Here is another example: lets assume an issue can be "closed" or "open" and the contract documents an action for opening or closing. Lets say the the action identifier for closing an issue is "close-issue". Your client GETs an issue and renders the title, description and so on for the end user. It also has button "Close" for closing the issue. Should the client brainlessly always enable that button and grab the "close-issue" action when the button is activated? No. It should look for the "close-issue" action first - and if the action is available then it can enable the button.

For what reasons should the "Close" button be enabled/disabled? From the client's point of view that doesn't matter as it relies on the server to inform it about the button state. This allows the server to evolve its business rules for making "Close" available on a issue. It can go from simply "If its open then it can be closed" to "If its open AND not locked by administrator AND not deleted THEN it can be closed".

As you touched upon yourself:

> have the interface to the resources be directly generated from the API responses - this is the case of HTML/Web Browsers, but it's just not how APIs are consumed.

Right. HTML + browser is in one extreme end of the spectrum where *everything* is in the response. In the other end we have hard coded clients with hard coded UI and hard coded business rules embedded in the client - such as for instance when to enable various buttons.

But its a spectrum. My example above shows how you can move some of the business rules out of the client and into the server. And you can do more - for instance include information about which input fields are enabled/disabled (and all the other stuff we know from HTML forms - except the actual visual layout).

These capabilities do of course have to be baked into the API contract. The client has to know that the server will tell it what is enabled and what is not.

Then you can ask yourself - is it worth it? Well, that "depends". For a beginning it depends the control you have over the clients. In the case of a mobile app. it is more less costless to upgrade clients in lock-step with the server. In which case it is probably not worth trading coupling for complexity. But its worth knowing what we can do and why we should, or should not, do it.

> But don't you need URI templates generally anyway? If you allow any sort of querying, you need to be able to construct a url based on some parameters.

Correct. Search for the link template "issue-query" in the issue tracker definition.

/Jørn

[Evan Cordell]

Why is that a cost? Why is "the way you use it explorable as well" an additional cost? Sorry I don't get that?

I was trying to illustrate that with the js client example: every api call you make must start at the root path, so that something simple like root.links.logo encompasses two separate API calls, with the possibility the client can't even figure out what the second should be (the case where is:logo is removed or changed).

Allow me to disagree with this. The contract is either in a domain specific media type or a generic one plus some sort of "profile" document (like my issue tracker description). That contract (the media type/profile) can/should specify which link relations a client can expect to find in the various responses.

Right, the contract that a web API provides is different from that which a local API provides. So what I'm wondering is: what can be done to bring them closer together? Is there a body of work that's attempting to do that? (The Web-API style that Jerome linked to seems like it's in that vein) Where can I read about approaches to this problem other than REST?

Everything else in your response makes sense and I agree, I'm just wondering about the alternatives. You get some great things from REST, but you also sacrifice some things (things that I think make for a good API). Can you get the benefits of both? 

[Evan Cordell]

Pedro:

I think Josh Bloch's argument about unchanging API's is about unchanging expectations. If as an axiom you accept that the capability of change is *part* of an API design, then suddenly all "change" forseeen by the API design is not really change per-se (i.e. a violation of expectations put forth by the API), it's just the API being what it is (i.e. effectively staying the same).

I mean, sure, we can say that if we expect an API to change then we're not surprised when it does change. But that doesn't really address how one actually develops against an API that could change at any time (I'd refer back to my example of a string class). 

The same for the rest of your description: I definitely see what you're saying, but short of going a full-browser route, I don't see how it could be relied upon for machine to machine communication (which seems to encompass any sort of user-initiated interaction short of a a full browser rendering of a hypermedia type).

 

If however the client doesn't understand the concept of "checking out" and has no capability to learn new concepts, then no amount of design can help it. The new concept and workflow change is effectively changing the clients expectations and thus introducing a breaking API change (for clients that are not capable of learning new concepts or don't understand the new concept).

I definitely agree here, which is kind of my point. The way one deals with these changes in a non-web API is to have a strong versioning system and giving the developer control of the version. It's really difficult to imagine a networked scenario that provides the same benefits (at least for me), but I also haven't seen much discussion of this aspect of web apis, and I'm interested in reading about approaches here, maybe doing some research (or being shown that the ideal is impossible). Does where I'm coming from make sense?

[Evan Cordell]

First off, it's more than possible that you're facing problems in 
which the full-blown REST style is not applicable.

This is very possible! The problem is that, the more I think about it, the more I convince myself that the full-blown REST style is not really applicable to Web APIs in general. I'd love to be convinced otherwise, or pointed to alternatives to REST that retain as many useful properties of REST as possible (as Web-API Style seems to be attempting). A lot of what seems to "work" about Web APIs right now is that they either a) don't change much or at all and b) die out pretty quickly. Things like hypermedia affordances seem like they don't make much of a contribution to API longevity compared to those.

Re: different amounts of hypermedia, I don't really see anything in Mike Amundsen's H-Factor description of media types that really seems problematic. What seems really difficult (which I've alluded to elsewhere) is having a format that supports changes in the scope/structure/behavior of the problem domain - like requiring or changing a step in a process - that can be handled in the client. 

Why is out-of-band documentation bad, but out-of-band assertions that api won't change okay? Is there no technical solution to this? 

[Steve Klabnik]

> every api call you make must start at the root path,

This is not really true, exactly. You only need to start at the root
path when you want to begin with no application state whatsoever.
That's why it's "hypermedia as the engine of application state":
hypermedia drives the state along. Doesn't mean you must start over
every single time.

[Evan Cordell]

That's a really good point, and I think I was being a bit more negative about it than necessary. 

Especially if a web API has a mechanism for instructing clients to start at the root path, then it doesn't even matter if an API changes during a client session (it still doesn't fix the other problems, but it does make the API client a bit less chatty)

--
You received this message because you are subscribed to a topic in the Google Groups "API Craft" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/api-craft/7c1tyHv2VQ4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to api-craft+...@googlegroups.com.

Visit this group at http://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/groups/opt_out.

--

Evan Cordell

318.471.6759

[Jack Repenning]

On Feb 27, 2014, at 8:13 AM, Evan Cordell <cordel...@gmail.com> wrote:

1. An important, maybe the most important, feature of a good API is that it does not change (especially while you are using it).

2. REST/Hypermedia intentionally address the problems of a changing interface

And that therefore they are in some ways "spiritually at odds" with one another.  

I think these statements are not as thoroughly at  odds as you suggest. In particular, however much we API designers might assert that APIs must not change, they none the less eventually do. The questions are all about (1) balancing the API stability need against the “requirements changed” need, and (2) making the change in the least-disruptive way possible. And maybe (3) choosing where to put your most emphatic words, in order to encourage good practice.

The “feature of good API … that it does not change” probably  implies “rather, when requirements change, a new API is created.”  The articulators of the REST zeitgeist accept that some change is inevitable, and rather than trying to ignore it or resist it until some Higher Power intervenes, they try to give a little more shape both to just exactly when “inevitable” really is, and to “how do we accommodate this?”

Whenever you see references to “versioning an API,” you *should* choke on your coffee and howl at the moon. This is proper design prejudice. But after cleaning up, and stepping back inside, and pushing back as hard as  you can on whatever force is trying to make you change, and losing sleep trying to conceive a way to meet the new (hopefully internal) goals without changing the externalities, you frankly sometimes can’t avoid  it. *That’s* when legitimate discussion of “versioning” begins.

In "Versioning Doesn't Work”, you present an excellent catalog of the difficulties of versioning any API, but these are not particular to REST, or even Web, work. This list of difficulties is a handy thing to keep around, as cautionary for yourself and ammunition in the “is this change really necessary?” argument. But even when proceeding with caution, changes sometimes are necessary. That’s when “versioning” becomes a topic (either in-place, or through a whole new API).

Jack Repenning

Socrates pointed out that writing is a tool to permit you to forget things. Nowadays, we have computers to forget stuff for you.

[Evan Cordell]

you present an excellent catalog of the difficulties of versioning any API, but these are not particular to REST, or even Web, work.

Definitely not - but in non-web work, you can just decide not to change the version of the API you're consuming.

I'd like to thank everyone that weighed in on this. It's been very helpful and has led me to lots of great reading and, I think, a better understanding of the problems surrounding web api craft. I've no longer lost faith in the ability of hypermedia to improve web APIs! It's hard to point to any one thing that anyone said, this conversation has just been generally useful to me.

If you're willing to bear with me, I'd like to outline my understanding of Current State of Things as a lead up to a request for pointers to reading. (I wrote this psuedo-formally but rather non-rigorously because it helped me to think about it like this).

Consider a server S with resources R1...RN and state transitions t1...tm. I'll call a client 'full' if it is capable of accessing the state of every resource (that it should be allowed to) and initiating every state transition (that is available at the time).

The following changes to S are possible:

- A resource Rn may be created, modified (added to, subtracted from), or destroyed.

- A state transition may be created or destroyed.

It seems clear to me now that if a change occurs to S, then assuming a sufficiently descriptive hypermedia transfer format, a client could be written to remain full.

Let A be a consumer of the client. How can A deal with a changing client?

- If resources are created, or only added to, A could be made to still function as normal.

- If state transitions are created, A could be made to still function as normal.

But A cannot handle the following changes:

- A resource is deleted

- A resource is subtracted from

- A state transition is deleted

(If A relies on any of the deleted information)

Versioning Solution:

Let RS be the current aggregate state of all Rn and tn in S. If a breaking change occurs, store RS and make it available under a version identifier.

Contract Solution:

Breaking changes are never made.

If a breaking change must occur, there must also be a mapping between RS1 and RS2. (for example, a deleted resource is still accessible, but data is stale or old, or aggregated from other resources. Or a single state transition that has been replaced by two smaller state transitions still functions)

Any change is communicated (e.g. headers). Gives time to consumers to change their use. 

Other?:

What other ways could a client consumer avoid breaking when a resource is deleted, changed, or when a state transition is destroyed?

So to summarize all of this: I was very concerned that there doesn't seem to be a good way to handle the types of changes that I've listed as breaking. Is there any current research/thought about addressing that? I'd love to read some ideas.

Thanks for the discussion, and I hope I didn't come off as too naive and stubborn, 

Evan

[mca]

"Let A be a consumer of the client."

I don't understand this sentence. 

Let A be a client?

Let A be a client of the client?

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Evan Cordell]

I did warn that it was non-rigorous ;)

I meant that there would be a generic client (say, that understands Cj or HAL), which would then be used by an application. Simply making the distinction between understanding the changes in the response (the client) versus understanding the changes to the semantic meaning of the response (the application). Apologies if there's a better way to make that distinction.

[Jack Repenning]

On Feb 28, 2014, at 3:50 PM, Evan Cordell <cordel...@gmail.com> wrote:

Let A be a consumer of the client. How can A deal with a changing client?

- If resources are created, or only added to, A could be made to still function as normal.

- If state transitions are created, A could be made to still function as normal.

*If* the new state transitions are optional / side-effect free.

To explore a popular example, suppose your resource is a Unix VM in some cloud. Such systems commonly expose two commands that both move the state from “running” to “stopped.” One performs clean-up before halting, which is safe but can sometimes hang (usually called “shutdown”). The other does no cleanup, which does no cleanup, making likely some problems on restart but virtually guaranteeing safety from hanging (usually called  “halt”).

Suppose you initially take the parsimonious path of only implementing the “safe” transition. Suppose, further, you foolishly name it “halt.” Your users may grumble at your naming, but they know how to work with your API.

Conscience strikes. You add the “forcibly unsafe” transition. Further, you see the error of your name-choosing ways: your new “halt” goes down dirty, as it should; your new “shutdown” does clean-up first.

Existing programs calling “halt” are surreptitiously switched from “safe” behavior to “unsafe.” Ick! That’s bad. Or “incompatible,” as we say. You "should not make such a change in any API.” You should, rather, treat it as a new (version of the) API.

-- 
Jack Repenning
Repenni...@gmail.com

[Evan Cordell]

The way I was thinking about it, that would constitute "destroying" a transition, and creating two new ones (one of which happens to be named the same thing). I see your point though. "Modifying" an existing transition could be either breaking or non-breaking. 

[mca]

"I meant that there would be a generic client (say, that understands Cj or HAL), which would then be used by an application" 

"a client used by an application" is an interesting POV; 

anyway... I think the "tell" here is this:

<snip>

But A cannot handle the following changes:

- A resource is deleted

- A resource is subtracted from

- A state transition is deleted

(If A relies on any of the deleted information)

</snip>

Is the above true for "web browsers"? No. Why? Because web browsers (which ARE clients) are designed as state machines. They DO NOT follow a per-determined path through a set of memorized resources with static relationships between each resource. They are built on a very different assumption. That the resource responses will always contain useful information and that information has the potential to lead to another (as yet never known before) resource. 

One of the reasons that hypermedia responses seem unhelpful to client-app builders may be that these app-builders are not interested in building apps that operate on this different assumption. Instead they want to build one-off, bespoke clients that have memorized all the possible paths from A-Z and are not at all interested in discovering as yet never known before resources.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Jack Repenning]

On Feb 28, 2014, at 4:13 PM, mca <m...@amundsen.com> wrote:

(If A relies on any of the deleted information)

</snip>

Is the above true for "web browsers"? No. Why? Because web browsers (which ARE clients) are designed as state machines. They DO NOT follow a per-determined path through a set of memorized resources with static relationships between each resource. They are built on a very different assumption. That the resource responses will always contain useful information and that information has the potential to lead to another (as yet never known before) resource. 

It appears to me that you are neglecting the human component of the client. To the extent that web browsers are free to follow the changing offered suite of possible next steps, it is because a human puzzles out the changes. Web designers have some freedom to change their apps at whim, because they can rely  upon the enslaved human to figure the changes out. (We all, of course, have a trove of experiences where we could *not* follow the changes, or discover the new path to the desired effect, or just plain felt we had better things to do. Even web apps are not so free as one might hope.)

One of the reasons that hypermedia responses seem unhelpful to client-app builders may be that these app-builders are not interested in building apps that operate on this different assumption. Instead they want to build one-off, bespoke clients that have memorized all the possible paths from A-Z and are not at all interested in discovering as yet never known before resources.

For client apps, the “human in the loop” is the app developer. But this human has a very different position in the system. If an API changes in a breaking way, the developer is not on the spot to respond differently. Instead, the canned, autonomous app simply fails. The developer learns of the change only when  the end user complains, or when a pager goes off, or by some other highly undesirable channel. Even if the new API still provides a way to achieve the desired effect, the client automaton probably can’t discover it in the heat of battle. Both the standard for what constitutes breakage, and the doctrines on how to handle it, must be significantly more rigid. 

I don’t know about client-app builders in general, but it’s this need to work without my pager going off that principally makes me hesitate over the popular idea of modeling APIs on web applications. 

-- 
Jack Repenning
Repenni...@gmail.com

[mca]

"It appears to me that you are neglecting the human component of the client. "

Yes, it may appear that way. But, I am not. 

First, of the work I currently do (or consult on) close to 80% of the client-code is directly human-driven. So the simple description I gave here is quite applicable. Second, for the remaining cases where the human is (to put a fine point on it) at a greater distance at runtime, the model outlined by Fielding and loosely described by me still works just fine. 

NOTE: it could very well be that my 80/20 experience it not at all yours or not even typical for those on this list. I cannot say. But that's what it is for me.

Now, when it comes to writing for the 20%...

The approach I take is to use a message model that is tuned for machines, not humans. While you *can* use HTML for machines, it's important to cut the choices to a very small set. Many of the M2M cases I implement allow machines only two hypermedia choices in each response: "Next" or "Stop" (not even "Back"!). Quite a bit of work can be accomplished in this way when you start treating machines as "users" and account for their "limited abilities."

I usually even cut back on the number of network operations allowed for machines. In the case of HTTP, I usually do not support POST since it is unsafe & non-idempotent - very hard to re-try failures.

When NOT using HTML (or Cj/Siren/HAL and other generic message models) a very successful approach is to design a domain-specific message model; one that is much more integrated into the domain problem. This makes is possible to bake more sophisticated decision algorithms into the client (but it also limits the _reach_ of both the client and the design; a trade-off). A trivial example of this is the Maze+XML message model I designed in order to allow people to build increasingly sophisticated "maze bots" to run maze of variying size and complexity.

Essentially, when you want to start "talking" to machines, you need to stop using "human languages" and need to start "talking on the machines' level." This is not at all complex, but it does take more work. 

Hopefully this fills in some of the things you saw missing in my posts.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Jack Repenning]

On Feb 28, 2014, at 4:53 PM, mca <m...@amundsen.com> wrote:

Essentially, when you want to start "talking" to machines, you need to stop using "human languages" and need to start "talking on the machines' level." This is not at all complex, but it does take more work. 

Which would seem to bring us back quite close to the OP’s

REST (hypermedia in particular) seems to be fundamentally at odds with API Design

… if the API in question is for machine consumption rather than human.

For the record, my 80/20 experience is darned close to 100% “machine-consumed.” Maybe we should all wear little Pareto badges! :)

-- 
Jack Repenning
Repenni...@gmail.com

[mca]

first, +1 on the badges! ;)

also, my point is that M2M scenarios take *more* design effort, not less. and whether the interface relies on hypermedia, configuration systems, stand-alone build-time models, etc. there is still an API Design effort. Whether the implementation style is RPC, MVC, MVVM, REST, Event-Base Notification, etc. they all rely in design of the interface.

I just don't see where REST (Hypermedia) is at odds with the design of *application* *programming* *interfaces*

feel free to talk me down.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Evan Cordell]

I suspect we might disagree on what an M2M interface is. I'm including the scenarios described by Jørn in this thread. (I think this is where Jack is coming from as well, but only he can say for sure).

It sounds like when you say M2M you mean completely autonomous M2M communication (as in your miss-grants-controller example). If I'm understanding your views from what you've written so far, I think you'd argue that if a human is driving the interaction at any stage, it isn't really M2M?

I think its useful to consider scenarios A-C in the above link to also be m2m - the user interacts with one machine, which then communicates with another machine (this is true of D as well, I know, and of web browsers in general, but in those cases presentation information is transmitted as well; in the others, it's necessary to have an interface local to the client that communicates with the API). I know you think these are probably "one-off, bespoke clients that have memorized all the possible paths from A-Z and are not at all interested in discovering as yet never known before resources." But that could include any HCI with a service in which the API is in any way abstracted from the user.

If you start to define an entirely dynamic interface for an API, you've written a (domain-specific and platform-specific) web browser. If you define an interface that is anything less than completely responsive to changes in the API, then you're susceptible to the breaking changes I described above.

I would also add that basically any client code that does anything interesting with the API aside from present it, you're in the same boat. Right?

I guess my view boils down to: a true REST/hypermedia client is really a form of HCI, rather than an API (an AHI - Application Human Interface, if you will).

Thoughts?

[mca]

"If you start to define an entirely dynamic interface for an API, you've written a (domain-specific and platform-specific) web browser."

"entirely dynamic" == "web browser" for versions of "entirely" of what? and "web" browser? so you assume work on only HTTP?

 

and

 

"If you define an interface that is anything less than completely responsive to changes in the API, then you're susceptible to the breaking changes I described above."

"anything less than completely responsive to changes" yep, this is a true statement.

but neither of these are very helpful POVs, IMO. these statements will always be true not matter what you are attempting or using in the attempt.

is that what you really want to settle on? that you are only interested in talking about "entirely dynamic", "web browser", and "anything less than completely responsive to changes"?

is that what you think I was talking about? in fact, I am not attempting nor even interested in those extremes. 

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Evan Cordell]

"entirely dynamic" == "web browser" for versions of "entirely" of what? and "web" browser? so you assume work on only HTTP?

I was using http language, but I didn't think my points were protocol-specific. I think I've done a poor job of explaining myself, let me rephrase.

 - If an API is anything aside from completely autonomous m2m communication, then a human must come into the picture at some point.

 - If a human needs to interact with an API, there are a few different levels of abstraction at which this interaction could come (I'll also just assume the interface between the human and API is graphical, though there are other examples). I thought Jorn did a good job of describing some of these in the thread I linked to.

 - The the only level at which your API can not break is the case when the user interacts directly with it.

   - If the message format of your API doesn't include presentation information, the user is in for a pretty bad UX (essentially, listing data and affordances)

   - If you include presentation information, you've written the equivalent of a web browser (if the protocol is http and the transfer format is html, then you have written a web browser)

   - At this point, it's difficult to call it an "API" because nothing is "programming" against it. And it's not really that label that matters, it's the way in which you can use it that does.

but neither of these are very helpful POVs, IMO. these statements will always be true not matter what you are attempting or using in the attempt.

is that what you really want to settle on? that you are only interested in talking about "entirely dynamic", "web browser", and "anything less than completely responsive to changes"?

is that what you think I was talking about? in fact, I am not attempting nor even interested in those extremes

Well, I'm interested in how one can expose a web service through an API such that a) that API can be "programmed against" and b) that API can evolve over time without breaking clients. Hypermedia is generally touted as a solution to at least the latter, sometimes both. 

I'm saying that with REST/Hypermedia you can evolve the API however you want only if you remove the ability to "program against" it. You could argue that that's not a guarantee web APIs can make, but then that's how people want to use most web APIs, so isn't it a problem that should be addressed?

If you talk about an API to a local service, it's easy to make both of those guarantees (that it can be used programatically and also change with time) because of versioning, and the ability of the developer to control the version in use.

In a network provided service, it's harder to make those guarantees. It seems like if you keep one, you lose the other. 

[mca]

<snip>

I'm saying that with REST/Hypermedia you can evolve the API however you want only if you remove the ability to "program against" it.

</snip>

"however you want" - sure. but again, why add these extreme phrases to your assertion? 

do you think that is what I am advocating ("however you want")? because that is not what i said, nor what i mean.

As for "remove the ability to program against" this assertion is not true in my experience -- unless your definition of "program against" is diff than mine.

- I can certainly conjure up scenarios where i can change the server in such a way as to ensure client failure. 

- I can also come up with ways to code a client app such that the slightest change in a server response will cause the client to crash.

But these are pathological POVs. I would never _plan_ to design fragile cases like this, would you?

I can also come up w/ cases where the opposite is true:

- servers can change in ways that do not cause clients to fail.

- clients are coded in ways that backward-compatible changes to server responses to no cause the client to fail.

Is this "easy"? no. Is this "valuable"? depends on several things. Is this "worth the added effort"? only in select cases.

Finally, IME, it is very easy to create circumstances that lead to failure.

It take more thought to start to come up with circumstances that lead to success. 

If you want to continue the convo, what should be the focus? the easy stuff? or the stuff that takes more effort?

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Evan Cordell]

"however you want" - sure. but again, why add these extreme phrases to your assertion? 
do you think that is what I am advocating ("however you want")? because that is not what i said, nor what i mean.

I didn't mean to be so extreme. I think there are certain evolutions of the resources/transitions that are both inevitable and require breaking any client (short of an API browser). I tried to outline those above, did you find it unconvincing? 

As for "remove the ability to program against" this assertion is not true in my experience -- unless your definition of "program against" is diff than mine.

- I can certainly conjure up scenarios where i can change the server in such a way as to ensure client failure. 

- I can also come up with ways to code a client app such that the slightest change in a server response will cause the client to crash.

But these are pathological POVs. I would never _plan_ to design fragile cases like this, would you?

I would definitely try to avoid those. But can one design a system such that the server never needs to change in a way that will cause client failure? 

I'd like to emphasize that I'm not asserting that hypermedia doesn't give you benefits.

I like REST in a lot of ways, and see its value. I see how the oft-overlooked hypermedia can help in many ways. I don't see that these things create an ideal API experience. I very much want to think about how to improve it. I don't know if there has been research in this area or not; I'm not sure how to catch up with the "state of the art" to think about this meaningfully. Is that the "stuff that takes more effort"?

I definitely see all of your points about my "pathological" POVs. But it's not so much about that, it's about wondering what could improve the "current situation"?

[Kijana Woodard]

Evan, I've been reading this thread with interest.

I don't see how the idyllic "local api" actually solves any of the problems you mention. A local api's safety relies on everything else in its vicinity remaining stable as well.

For instance, upgrading the OS can break a local api without the developer expecting it. Many of us have gotten panicked calls from OPS when they "performed a simple upgrade that shouldn't have caused a problem with the application".

[Felipe Sere]

Hey!

So I finally find time to pitch in :)

I think a lot of the discussion circles around “control”, especially “control over change”.

In my experience, that is something you automatically loose once whatever you use is behind a network card.

You simply can not assume that “what works locally on my machine on a single processor with a single memory model” works across the network. You get network failures and other kinds of disasters.

Hence you have to embrace change in one way or another. The pessimistic approach is to try to force things to never change. I get the vibe that is where you are at, Evan. Though this might be doable in corporate environments, its more of an organisation issue rather than a technical one.

One the other hand is explicitly allowing for change, in a  controlled manner. This is where I see Mike A, right? I personally agree with this view. Build systems that allow certain changes without breaking. This is probably just as hard trying to force things to never change, but it gives you a certain degree of freedom. You can then use that to “wiggle” your way out of a bad situation.

In a RESTful world you’d achieve a certain degree of freedom by a) Build on link relations and links c) working with affordances that tell the client what he can/should do. This gives you the freedom to change URLs and add new steps which newer clients will understand.

Ok… my 5min just ran out again :)

Cheers!

Felipe

[Paul Cohen]

Hi,

On Mon, Mar 3, 2014 at 12:55 PM, Felipe Sere <felip...@gmail.com> wrote:
> I think a lot of the discussion circles around "control", especially
> "control over change".
> In my experience, that is something you automatically loose once whatever
> you use is behind a network card.
> You simply can not assume that "what works locally on my machine on a single
> processor with a single memory model" works across the network. You get
> network failures and other kinds of disasters.
>
> Hence you have to embrace change in one way or another. The pessimistic
> approach is to try to force things to never change. I get the vibe that is
> where you are at, Evan. Though this might be doable in corporate
> environments, its more of an organisation issue rather than a technical one.
>
> One the other hand is explicitly allowing for change, in a controlled
> manner. This is where I see Mike A, right? I personally agree with this
> view. Build systems that allow certain changes without breaking. This is
> probably just as hard trying to force things to never change, but it gives
> you a certain degree of freedom. You can then use that to "wiggle" your way
> out of a bad situation.
>
> In a RESTful world you'd achieve a certain degree of freedom by a) Build on
> link relations and links c) working with affordances that tell the client
> what he can/should do. This gives you the freedom to change URLs and add new
> steps which newer clients will understand.

I think Felipe is spot on with this post.

I would like to emphasize the organisational - or rather business -
aspect what Felipe says. The API:s we are discussing are intended to
be used over organisational boundaries; furthermore they will
typically involve many (client) organisations. Those organisations may
well be corporate entities. In a corporate environment it is
impossible to require all development processes and planning in all
organisations to be lockstep integrated. That is the main reason for
why we must, as Felipe says, "embrace change in one way or another";
different organisations and corporate entities must be as free as
possible to evolve their products and systems according to their own
business processes, plans and requirements.

/Paul

--
Paul Cohen
www.seibostudios.se
mobile: +46 730 787 035
e-mail: paul....@seibostudios.se

[sune jakobsson]

So we can all agree to the "It depends" :)

Unfortunately REST is not that well understood, and still has its
caveats, but I still have issues explaining it to "my grand mother" on
why it is so much better.
And adding more acronyms to the soup does not really make it tastier.

Sune

[Evan Cordell]

I don't see how the idyllic "local api" actually solves any of the problems you mention. A local api's safety relies on everything else in its vicinity remaining stable as well.

You're absolutely correct and your example is apt. But in the web case, you have the same concerns in the client software, plus the server software, plus the issues I outlined. 

I just think it's worth thinking about. There might be some meat there, you know? I haven't seen any arguments for why it should be impossible to address or improve upon the issues we've been discussing. 

I think a lot of the discussion circles around “control”, especially “control over change”.

In my experience, that is something you automatically loose once whatever you use is behind a network card.

Definitely. But I'm not sure I'm so against change as you make it out; more than that I'm trying to explicitly define the ways in which changes affect clients in order to think more clearly about how those can be mitigated (beyond the solutions presented in REST, which, while good, do not fully address the problem.)

I'm starting to think that I'm doing a little too much "public pondering." If so, I apologize.

Here's the question that I'm focusing on: "Can a REST architecture be modified in some way (by adding, changing, or removing constraints) such that a REST API  and its clients are resilient to a larger subset of changes?"

And a follow up: "If the answer to the previous question is 'No,' can it be satisfactorily proven?"

It's very possible that I currently lack the knowledge to answer either of those. But they interest me, so I'd like to study them until I can at least answer them with something resembling confidence.

[Kijana Woodard]

When I attempt to respond to your question, I am stymied by the number of axes involved.

Care to shape it with some "Givens"?

My sense is that answering your question is one of the goals @mca seeks in the design of his media types.

[Mike Kelly]

On Mon, Mar 3, 2014 at 9:42 PM, Evan Cordell <cordel...@gmail.com> wrote:
> Definitely. But I'm not sure I'm so against change as you make it out; more
> than that I'm trying to explicitly define the ways in which changes affect
> clients in order to think more clearly about how those can be mitigated
> (beyond the solutions presented in REST, which, while good, do not fully
> address the problem.)

You can't "fully address" the problem of change in m2m APIs because
machines are inherently immalleable. Machines clients are written by
people... even if you came up with the perfect protocols for dealing
with change in your API, the odds of you being able to convey those
protocols unambiguously to the people who will write clients against
your service is pretty low. It's a lot more likely you'll just put
people off and they won't bother using your service at all because
it's too painful. But then again, I guess an API with 0 clients
doesn't have an evolvability problem...! :)

If it's not m2m, HTML should do the trick.

Cheers,
M

[Steve Klabnik]

This depends on what kind of change you mean, I think. Obviously "I
built a new service" isn't possible, but "This action is temporarily
unavailable" should be able to be handled by machines.

[Mike Kelly]

Right, as I said, you can't "fully address" the problem of change.

There's actually a lot machines can't do.

I think a lot of hypermedia promotion doesn't click with people
because they intuitively understand that a lot of these dynamic
affordances don't get you very far in APIs and just add a lot of
pointless complexity.

Let's think about an API control that allows the server to dynamically
control the method a machine client uses for a given request:

Suppose I just changed the method for a request from PUT to POST. That
means all the code that depends on that particular request being
idempotent just broke; either because it's clever enough to figure out
it can't do that any more and needs changing, or because it's not
clever and just carries on making non-idempotent requests as if they
were idempotent.

Cheers,
M

> --
> You received this message because you are subscribed to the Google Groups "API Craft" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/api-craft.
> For more options, visit https://groups.google.com/groups/opt_out.

--
Mike

http://twitter.com/mikekelly85
http://github.com/mikekelly
http://linkedin.com/in/mikekelly123

[mca]

"code that depends on that particular request being idempotent"

why write into your app a breaking dependency on idempotence? is this a requirement? or a "want to have"?

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Mike Kelly]

because code that leverages idempotency can be more efficient and
resilient? Presumably that's why bother to have that property distinct
between the two methods in question.

What's the solution?

you should never leverage idempotency in code or architecturally as a
client? or that you should always write code that conditionally
defends against idempotency being removed? or something else?

[mca]

some possible alternatives to writing code that is dependent on idempotency (usually this comes into ply for failed requests):

- always assume non-idempotency (worst case scenario)

- write code that refuses to execute write operations unless an idempotent write is available.

- write code that mimics idempotency on its own by using memorized responses from GET pre- and post-write

- write code that understands when a request is idempotent and acts accordingly 

there are proly others, but these come to mind quickly.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

[Mike Kelly]

On Mon, Mar 3, 2014 at 11:57 PM, mca <m...@amundsen.com> wrote:
> some possible alternatives to writing code that is dependent on idempotency
> (usually this comes into ply for failed requests):
> - always assume non-idempotency (worst case scenario)

That's going to create a lot of unnecessary complexity and/or inefficiency.

> - write code that refuses to execute write operations unless an idempotent
> write is available.

This is what I meant by a breaking change, since your code would no
longer be able to achieve it's goal.

> - write code that mimics idempotency on its own by using memorized responses
> from GET pre- and post-write
> - write code that understands when a request is idempotent and acts
> accordingly
>

I really struggle to buy into the idea that the average client developer will:

1. want to do this
2. bother to do this
3. implement it properly

and all we're talking about here is a change of method name. How often
do you really need to change that in practice? I think this is why we
don't "get it", because it seems like attempting this level of
dynamism in an m2m API just isn't worth it.

Cheers,
M

[mca]

yep, some of these are things i do not want to do. what i offer here are alternatives. people can make their own choices on whether these a) will even work, b) are worth the effort given the likelihood of the exepcted change.

you asked for some solutions, and i offered them. i suspect more are possible. i have no idea of any of them would be "worth it" to you.

 

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund

M

[Mike Kelly]

The specifics of the solutions or my opinion aren't really that
important here, I'm attempting to demonstrate why some of the common
hypermedia API wisdom (particularly around forms in m2m APIs) doesn't
click with people.

Cheers,
M

[mca]

ok. cheers.

mca

+1.859.757.1449
skype:mca.amundsen
http://amundsen.com/blog/
http://twitter.com/mamund
https://github.com/mamund
http://linkedin.com/in/mamund