Website as API

[Kijana Woodard]

On a recent thread[1], Ruben Verborgh asserted that "...in the longterm, we should convince developers to just provide machine-readable representations of the resources they offer through their websites".

I've been on the other side of that debate for a few years, but Ruben's arguments on his blog [2] are compelling enough to get me to reconsider my position.

I rest my stance on two ideas.
- Websites change at different rates than APIs.

- Websites change for different reasons than APIs.

Coupling the two is problematic.

For certain kinds of sites, like the info sites mentioned in the blog post, this approach may make sense. I don't think there is "One True Way".

However, lets take a simple blog site as an example. For a post, you have the post, but you also have navigation to next articles, comments, tag clouds, archive links, roll call, ads, etc, etc. Which of these things do you get for the json representation? What heuristic do you follow to decide?

I'm perfectly happy is there is asymmetry between representations. It's just not clear how to make expectations predictable to a programmer. 

Say you have /posts/123 which reveals /posts/123.json and /posts/123.html as "alternate representations".

Is it "ok" if GET /posts/123 Accept: text/html returns something different from GET /posts/123.html?

I think I can live with any answer here.

The "reasons for change" argument carries more weight.

If a business person says "I want to change the blog so that all posts infinite scroll on the home page" and my response is "but that will break our API clients", that conversation isn't going to go well.[3] Humans are resilient to change and various stakeholders like to "play" with the website in ways that may not be conducive to a backwards compatible api.

Following onto that, the pace of change leads to decisions that might be acceptable for a human centered experience, but disastrous for machine clients. There's probably little to no thought given to "compatibility for a client built a year ago" when someone wants to change a website. 

I consider myself persuadable on this topic.

Thoughts?

Alternate opinions from the two presented?

[1] https://groups.google.com/forum/#!topic/api-craft/7oD2W3qGSBQ

[2] http://ruben.verborgh.org/blog/2013/11/29/the-lie-of-the-api/

[3] Ignore whether or not this is a bad idea. The reasons for or against doing a website design change shouldn't ride on api client compatibility.

[Ruben Verborgh]

Hi Kijana,

Interesting topic. I'm especially curious about that second argument, that websites would change for different reasons than APIs.

However, I think it's important for that second case to come up with a few strong examples, so we can see more general patterns.
The infinite scrolling one is countered rather easily.
(One solution: in the background, the infinite paging mechanism likely still fetches pages.)

Best,

Ruben

[Kijana Woodard]

Stipulation: Working with an invented example is problematic. :-D

Back to the blog.

For humans [html], they want to see a post and comments together as "one thing".

For an API, maybe it's better to make those two things [or N things if each comment is addressable].

Making a new decision for humans shouldn't lead to a breaking change for api clients.

This also brings up context.

Perhaps the website is for *consuming* the blog while the api is for *writing* posts. There is a "post", but there's only a thin overlap between what the website would provide and what the api would provide. You could even question if they are the same system: browsing, sharing vs drafting, editing, reviewing, publishing.

I'd also say there's a difference between an api that is going to, ultimately, allow a client to build an alternate human interface and one that is supposed to programmatically manipulate resources according to coded rules. As I've gone down that path before, simply thinking through what it would mean to have a program automatically manipulate resources starts to push hard against the underlying business model.

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to api-craft+...@googlegroups.com.
Visit this group at https://groups.google.com/group/api-craft.
For more options, visit https://groups.google.com/d/optout.

[Ruben Verborgh]

> For humans [html], they want to see a post and comments together as "one thing".
> For an API, maybe it's better to make those two things [or N things if each comment is addressable].

Alright, so we are indeed looking for examples where humans and machines consume different resources.

The above scenario I have encountered a couple of times.
This is also quite straightforward:
not every resource needs to have its own representation in all supported representation formats.

So if the human-readable version has no unique representation for comments,
just redirect to the blog post.

> Making a new decision for humans shouldn't lead to a breaking change for api clients.

Indeed.

> Perhaps the website is for *consuming* the blog while the api is for *writing* posts.

Fair enough. Not all actors should have a representation of everything.

Best,

Ruben

PS Not trying to push my own view here,
but challenging people to find examples :-)

[Kijana Woodard]

"Unfortunately, you’ve been lied to. Or more precisely, you got what you asked for. Because I bet that you were the one who ultimately decided: yes, I need an api!  No, you don’t. It’s the same content, why would you need two interfaces..." [1]

I guess my assertion, with regard to the quoted text, boils down to "maybe, maybe not". Sure, you might have a website that is amenable to also being an API. Once you make that choice, you'll have to more rigorously consider the changes you wish to make than you did before.

What I've experienced is that companies decide "we need an api" to tick off a marketing box. The depth of thinking is usually along the lines of "the api should be able to do whatever you can do on the website". That leads to the next idea of "maybe we should build the website with the same api" [ditto for internal api is "the same" as the external api]. 

My recommendation for them is to do nothing. That usually doesn't go down well. ;-)

My point is that they haven't really begun to consider the implications of having a programmable interface into their business. There are affordances that arise for api clients that don't make sense for humans navigating a site and vice versa. Further, they generally aren't used to thinking in terms of backwards [and forwards] compatibility or about consumers with heterogeneous capabilities. Generally the goal is "to get something up quickly" which is at odds with something that is well designed to survive continuous evolution over a long time frame.

So my next recommendation would be....build a separate api [and start frighteningly small] so at least you reduce the number and complexity of the design considerations.

[1] http://ruben.verborgh.org/blog/2013/11/29/the-lie-of-the-api/ - p.s. isn't it silly that we can't quote directly into any html page?

[Michael Tiller]

Just a short comment.

I've been thinking about this same issue for a project I'm working on.  I don't want to go into too much detail, but you can think of it as being similar to a content management system.

One thing that I suspect may be an implicit assumption here is that the server is generating both the html and the JSON.  But I've taken a different approach.  In my case the API can generate both HTML and JSON, but the HTML isn't actually human readable HTML.  What it does is launch code that then queries the (Siren) API to construct the HTML client side.  The point is that this code that gets launched is smart about the content.  It communicates through the API and transforms the representation on the client side.  But it could be done on the server side, I guess.

I chose to apply this to this project because I thought it would work well.  I'm not convinced it is a general solution.  But one of the issues for me was recognizing that a hypermedia format can actually offload a lot of the logic of the application and focus on simply presenting a good view of the underlying content.  In this sense, lots of the navigation logic isn't necessary because you are actually navigating using the browser, jumping from (hypermedia) resource to resource rather than having to map out "routes" in your application.

It is still a little experimental, but so far I've been pretty happy with how it has been working out.  I think it will be easier to program it this way and I have some ideas on how to make the whole thing modular and easily extensible.  Unfortunately, I haven't written anything up on it yet and I don't have any concrete examples.  So a lot of this is just arm waving.  But since the topic came up, I just thought I'd mention it.

--

Mike

[Mike Kelly]

_____________________________
From: Kijana Woodard <kijana....@gmail.com>
Sent: Thursday, April 7, 2016 10:40 pm
Subject: [api-craft] Website as API
To: <api-...@googlegroups.com>

On a recent thread[1], Ruben Verborgh asserted that "...in the longterm, we should convince developers to just provide machine-readable representations of the resources they offer through their websites".

Developers should pick a design that produces the outcome they want. For most the target outcome will relate to growth.

In other words, most developers will be optimising their API developer experience to reduce the barriers and costs for third parties integrating with them.

HTML is more inconvenient for developers to work with than machine readable formats like JSON.

I suspect this is the reason APIs aren't exposed as HTML, rather than a lack of evangelism or education on the matter.

[Kijana Woodard]

Fwiw, it's always struck me as odd [ironic? unfortunate?] that a javascript client running inside a web browser has difficulty working with html.

--

[Johan Groenen]

I don't need your web page as an API endpoint, I need the (multiple) data presented on your page to be available in machine readable format. Webpages usually combine multiple data streams into one view for the user, and also add navigation features for that specific site.

So don't make a single API for your site, but make sure all the data on your site are available through api's (so I don't have to scrape your website).

[Johan Groenen]

JavaScript works fine with HTML (DOM), thing is that DOM is not meant for data storage and retrieval, but for data presentation.