API design - Ease of development THE primary driver

[Gabor]

Hi guys,

I'm new to this forum, so please bear with me if I ask for your opinion on a topic that's been already discussed here.

Let me direct your attention to an apigee video that I loved watching: http://www.youtube.com/watch?v=QpAhXa12xvU

Brian and I started a discussion (yes, on YouTube :() on how much we should use HTTP headers in favor of URL params. The context was something like how output formats (json, xml, etc) should be specified and I raised the idea of relying on Accept header. Brian's response was that it's easier to debug your app if you can specify this value in the URL, because then you can simply use a browser and don't need to bother with a web debugger tool, like Fiddler. My opinion is different and since I couldn't finish my post on YouTube, let's discuss it over here.

First, I know it's not the biggest problem on Earth. Yet, my biggest worry is not how this technical problem can be solved, but how much we should cater for app devs vs how flexible API design we work out and will rely on service implementation side. Suppose we have a requirement of enabling API users to specify a priority-ordered list of possible output formats. The two solutions would be:

- (Brian) ?format=(json,xml)

- (Gabor) Accept: application/json,application/xml

First question: is it common to introduce functionality at URL-level that is already supported by HTTP?

Second question: is catering for app devs at any price really the only primary requirement for the API as a product? I mean, my example might not be the best illustration, but I can imagine a situation where two solutions contradict with one another and that one would win which would make app devs' life easier even if it's painful for service implementors.

Any opinion?

Gabor

[Mike Kelly]

There's usually a happy middle ground.. you can use both approaches
together in various ways, e.g. you can include a Content-Location
header in negotiated responses that tells the client the specific URI
of the representation the server is responding with. i.e:

GET /foo
Accept: application/hal+json

200 OK
Content-Location: /foo?format=hal+json
{
....

> --
> You received this message because you are subscribed to the Google Groups
> "API Craft" group.
> To unsubscribe from this group, send email to
> api-craft+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/api-craft?hl=en.
>
>



--
Mike

http://twitter.com/mikekelly85
http://github.com/mikekelly
http://linkedin.com/in/mikekelly123

[Jørn Wildt]

Some server frameworks, like OpenRasta, allows you to configure the URL extension that indicates a certain content-type, giving you both solutions almost for free.

And, yes, it is nice to be able to use a browser to, well, browse through the data without having to resort to using Fiddler.

/Jørn

[Peter Monks]

Big +1 from me for what Jørn said.  Allowing clients to request /foo.xml or /foo.json (or whatever) as they wish seems to me to be both more natural & approachable to the developer, while also allowing a simpler and more nicely encapsulated server implementation.

The one possible advantage of the header approach is that it provides a fallback (negotiation) mechanism, which extensions don't.  But there are other ways to skin that cat i.e. providing an API discovery mechanism (HAL or OPTIONS requests or WADL or whatever) that return the available response types for each resource.

Cheers,
Peter

 

[Mike Schinkel]

Make that a 2nd +1 to what Jørn and Peter said.

I too much prefer using extensions to indicate data types, and I prefer to avoid headers unless absolutely required.

-Mike

[Gabor]

Ok, guys, got it. Thanks for the response!

I must admit I wasn't familiar with HAL so thanks for sharing info on it. Nevertheless, it's a new stuff in contrast with HTTP that everybody understands and supports, right? Don't get me wrong, I'm absolutely not against new things (on the contrary), but why working out workarounds when existing built-in support is available? On the other hand, I like HAL's flexibility to address resources at different locations, for example.

Anyway, your feedback is invaluable to me. My point wasn't necessary particulary on this topic, but that catering for app devs above all - and I don't agree with that. I can see that we're programming for app devs who are a very important member of the whole value chain, but we're equally programming for the whole system. App devs, and eventually their consumers, wouldn't be happy if the underlying implementation of an API would not perform at best scale just because they're tied due to some compromise we made at API-level.

Cheers,

Gabor

[Peter Williams]

I think having media type specific URIs is an useful debugging
feature. However, for actual working code i strongly prefer using the
accept header. It allows for more evolvability on the client side.
(eg, Clients can switch from xml to json and still use all their
bookmarks.) It also allows integration of clients that do not support
the same media types.

For example, imagine some hal+xml specific URI X on a server which is
able to produce both hal+xml and json-ld representations of the
conceptual entity X represents. Program A which likes hal+xml acquires
URI X, fetches it and does some work. Program A then hands URI X to
program B which doesn't understand hal+xml but does understand
json-ld. Since we are using a representation specific URI the
integration get pretty complicated (read: probably dead) at that
point. If we were using a URI that represent the conceptual entity,
rather than a specific representation of that resource, each client
would request the representation it needs to work and the integration
would be quite straight forward.

Peter
barelyenough.org

[Peter Monks]

Maybe I'm being naive, but isn't switching the extension just as easy as changing a header? Provided the server exposes some kind of self-discovery mechanism, program B can determine for itself whether the server has a more suitable representation available than the one program A provided to it.

Cheers,
Peter

[Steve Klabnik]

There are a few browser extensions that make working with headers fairly trivial.

[Peter Williams]

On Wed, Sep 12, 2012 at 12:20 PM, Peter Monks <pmo...@alfresco.com> wrote:
> Maybe I'm being naive, but isn't switching the extension just as easy
> as changing a header?

As a client it is not at all easy to "switch extension". Servers
handle format selection in the URI in a lot of different ways. Some
use URIs that end in a file extension type thing (eg, `.xml`). Some
use a name and value in the query string (eg, `?format=xml`). Other
put the format selector early in the path part of the URI (eg,
`/v2/projects`). Even if, as a client, you can identify the format
selector in the URI, you still won't know what to replace it with.
Short format identifiers are entirely arbitrary and defined by the
origin server. How do clients know if the server in question has
decided to call json-ld representations `jld`, `json-ld`, `json` or
something truly random.

Media types, otoh, are well standardized and so is the accept header.
As a client i always know exactly which media types i can work with
and i always know how to put that info in the Accept header.

> Provided the server exposes some kind of self-discovery mechanism,
>program B can determine for itself whether the server has a more
> suitable representation available than the one program A provided to it.

`alternate` links in the HTTP header are a reasonable way to implement
such a discovery mechanism. But forcing every client to check for more
useful alternates after every request strikes me as a substantially
worse than just implementing content negotiation as defined by HTTP.
Many web frameworks even make implementing content negotiation pretty
painless these days.

Peter
barelyenough.org

[Jørn Wildt]

+1 for that Pete

/Jørn

Peter
barelyenough.org

[Pat Cappelaere]

Peter,

In an hypermedia world, you want to follow links (or send them via email to your friends)…
So how do you send an email with a link to your developer friend and then you say: set the accept header to blah-blah to see that output…
I guess you are saying that links are not enough now…??? you can't just click… wouldn't this be disappointing?

Pat.

[Mike Schinkel]

On Sep 12, 2012, at 4:22 PM, Peter Williams <pe...@barelyenough.org> wrote:

> Even if, as a client, you can identify the format
> selector in the URI, you still won't know what to replace it with.
> Short format identifiers are entirely arbitrary and defined by the
> origin server. How do clients know if the server in question has
> decided to call json-ld representations `jld`, `json-ld`, `json` or
> something truly random.
>
> Media types, otoh, are well standardized and so is the accept header.
> As a client i always know exactly which media types i can work with
> and i always know how to put that info in the Accept header.

You make a good point about media-type standards.

However, I think you are paying a dollar to save a dime here (sorry for the US-centric currency metaphor.) Fully-descriptive URLs have huge value as others have explained, working towards a standard method for discovering what extensions map to what media types seems like a better approach to me; how hard would it be?

> `alternate` links in the HTTP header are a reasonable way to implement
> such a discovery mechanism. But forcing every client to check for more
> useful alternates after every request strikes me as a substantially
> worse than just implementing content negotiation as defined by HTTP.

Why would they need to check with "every" request? Why not just the first?

> Many web frameworks even make implementing content negotiation pretty
> painless these days.

Except that interacting with them then *requires* use said web frameworks vs. being able to explore an API with a simple browser. Too bad HTML doesn't have <form> <input>s where type="header" or similar.

-Mike

[Peter Williams]

On Wed, Sep 12, 2012 at 4:26 PM, Pat Cappelaere <cappe...@gmail.com> wrote:
> Peter,
>
> In an hypermedia world, you want to follow links (or send them via email to your friends)

Agreed.

> So how do you send an email with a link to your developer friend and
> then you say: set the accept header to blah-blah to see that output…
> I guess you are saying that links are not enough now…??? you can't
>just click… wouldn't this be disappointing?

Of course you can just click the link. Once you do, which ever program
actually handles that click should inform the server of what it need
to accomplish its goal. Not having the format in the URI actually
makes this type of integration *much* easier. If the format is
embedded in the URI the click handler's job is a lot more complicated
if it needs something different.

Representation specific URIs are helpful for the developer playing
around use case, but they are actively harmful for users who actually
want to accomplish something. The are also harmful for developers who
want to develop an easily evolvable and maintainable system.

Peter
barelyenough.org

[Patrice Cappelaere]

Bookmarks are not harmful, are they?

And they are so much easier :)

Pat/

-- 
Patrice Cappelaere

Sent with Sparrow

[Mike Schinkel]

On Sep 12, 2012, at 7:07 PM, Peter Williams <pe...@barelyenough.org> wrote:

Representation specific URIs are helpful for the developer playing
around use case, but they are actively harmful for users who actually
want to accomplish something. 

Can you explain with specifics how links are actively harmful for users?   And how users even recognize this alleged harm?

The are also harmful for developers who
want to develop an easily evolvable and maintainable system.

Same question for developers.

-Mike

[Peter Williams]

On Wed, Sep 12, 2012 at 5:14 PM, Patrice Cappelaere
<cappe...@gmail.com> wrote:
> Bookmarks are not harmful, are they?
> And they are so much easier :)

Bookmark are not harmful. In fact, they absolutely vital.

Peter
barelyenough.org

[Peter Williams]

On Wed, Sep 12, 2012 at 5:18 PM, Mike Schinkel <mikesc...@gmail.com> wrote:
>> On Sep 12, 2012, at 7:07 PM, Peter Williams <pe...@barelyenough.org> wrote:
>>
>> Representation specific URIs are helpful for the developer playing
>> around use case, but they are actively harmful for users who actually
>> want to accomplish something.
>
>
> Can you explain with specifics how links are actively harmful for users?
> And how users even recognize this alleged harm?

Well, if i have the URI for a xml version of a resource but i really
want the html version i am pretty much screwed. As developer i might
be able to guess the needed transformation but how are civilians
suppose to figure that out?

>> The are also harmful for developers who
>> want to develop an easily evolvable and maintainable system.
>
> Same question for developers.

The answer is basically the same. As a human and developer you might
be able to guess how to transform a URI for the xml representation of
an entity into the URI of the json representation of that entity but
building user agents with that level of pattern matching and intuition
is difficult. In the face of representation specific URIs user agents
require more complex implementations and more maintenance/human
intervention than they would if resources supported content
negotiation.

Peter
barelyenough.org

[Mike Schinkel]

On Sep 13, 2012, at 11:31 AM, Peter Williams <pe...@barelyenough.org> wrote:

Can you explain with specifics how links are actively harmful for users?
And how users even recognize this alleged harm?

Well, if i have the URI for a xml version of a resource but i really
want the html version i am pretty much screwed. As developer i might
be able to guess the needed transformation but how are civilians
suppose to figure that out?

You can't be serious?  It is exactly for the user that this makes sense.  A user sees this URL and wants an HTML version:

http://example.com/product-list.xml

Simple, replace extension "xml" with "html"; I've done this countless times on site's I'm surfing although for me it's usually "xml" vs. "json":

http://example.com/product-list.html

What am I missing that makes this so hard for a user?  Remember, users are resilient to errors.  If the URL for html is no extension or "htm" they can try that too. Probably the only reason this would be any issue for users is not enough sites doing it this way.

OTOH, how would a user send a different header to get a different representation?  As a developer, I don't even know how a user would do that.

Same question for developers.

The answer is basically the same. As a human and developer you might
be able to guess how to transform a URI for the xml representation of
an entity into the URI of the json representation of that entity but
building user agents with that level of pattern matching and intuition
is difficult. In the face of representation specific URIs user agents
require more complex implementations and more maintenance/human
intervention than they would if resources supported content
negotiation.

Again, how hard can this be?  A simple regex replacement can swap out extensions, in PHP this switches a URL to .json (Javascript is similarly easy but I can't do from memory):

$new_url = preg_replace( '#\.(xml|html|json)(\?.*)?$#', '.json', $url );

Now I will agree the client needs to know what translations the server supports, but if we are talking about an API client that is going to do something less trivial then simply browsing and downloading representations then the client will need some out-of-band information anyway. The following would be one simple way to provide that information via HTML pages on the site:

<meta name="media-types" content="xml=application/xml&json=application/json&html=text/html" />

Or this:

<link rel="media-types" href="/media-types.json" />

Where /media-types.json contains:

[ 

   "xml": "application/xml",

   "json": "application/json",

   "html": "text/html"

]

And yes, none of this is standard.  But the alternates are not standard either, so why not advocate for this as a simple standard that IMO works extremely well for users and could be a great standard for developers (and by "this" I mean URL extensions mapping to media types, not the specifics of my mapping examples; mine were just quick and unvetted examples.)

Simply put, using URL extensions to indicate media types is the simplest and most elegant approach and one that I thank the designers of an API for using.  Why prefer complexity when it's not needed?

-Mike

[Steve Klabnik]

The w3C has an excellent article that would be relevant to this discussion; it's about multiple languages, though. I can't seem to find it at the moment...

[Peter Williams]

On Thu, Sep 13, 2012 at 11:01 AM, Mike Schinkel <mikesc...@gmail.com> wrote:
> You can't be serious? It is exactly for the user that this makes sense. A
> user sees this URL and wants an HTML version:
>
> http://example.com/product-list.xml
>
> Simple, replace extension "xml" with "html"; I've done this countless times
> on site's I'm surfing although for me it's usually "xml" vs. "json":
>
> http://example.com/product-list.html
>
>
> What am I missing that makes this so hard for a user?

That is easy for you or i but i don't think my mom would know how to
do that. Are you actually suggesting it is an ok user experience for
users to *ever* have reconstruct URIs?

> Remember, users are
> resilient to errors. If the URL for html is no extension or "htm" they can
> try that too. Probably the only reason this would be any issue for users is
> not enough sites doing it this way.

User are resilient. Their strategy for recovering from a failure of
this sort is usually to find a competitor whose URIs just work.

> OTOH, how would a user send a different header to get a different
> representation?

Users have absolutely zero interest in this sort of thing. They just
give the uri to a user agent (usually by clicking it) and the user
agent handles all that administrivia. The application GETing (etc) the
uri is the thing that should decide what format is needed, not the
user.

> A simple regex replacement can swap out
> extensions, in PHP this switches a URL to .json (Javascript is similarly
> easy but I can't do from memory):
>
> $new_url = preg_replace( '#\.(xml|html|json)(\?.*)?$#', '.json', $url );

The list of all known file extensions is long a growing. Do you really
want to maintain that regexp? And that pattern ignores the servers the
don't use file extension but use a `?format=...` query string, or that
use an api version id in the uri, etc.

[snip...]
>
> And yes, none of this is standard. But the alternates are not standard either.

The alternate link rel is standardized[1], btw. If you are going to do
hyper media driven content negotiation please, please, please use
rel=alternate link headers. At least then i will not have to
find/write a parser for all the different formats that could be
returned and understand some custom reinvention of the wheel.

Even it you do use standard alternate links the client is going to be
more complicated. It is also going to have to, potentially at least,
make more requests against the server.

> Simply put, using URL extensions to indicate media types is the simplest and
> most elegant approach and one that I thank the designers of an API for
> using. Why prefer complexity when it's not needed?

Having exactly one representation for every resource may be simple but
it is far from elegant. It increases the number of requests clients
must make (clients must sometimes remake requests based on alternate
links). It makes identity harder to establish (is
"api.example.com/order/42.xml" the same order as
"example.com/order/42" or not?). It makes it harder to evolve clients
to new formats (if a client wants to move from some custom xml to
json-ld and it has a bunch of urls for xml representations bookmarked,
it must transform those URIs or support both xml and json-ld). It is
also harder on the server side because you have to be very disciplined
about providing the alternate links everywhere or clients are
potentially dead in the water.


[1]: http://tools.ietf.org/html/rfc5988#section-6.2.2

Peter
barelyenough.org

[Mike Schinkel]

On Sep 13, 2012, at 2:04 PM, Peter Williams <pe...@barelyenough.org> wrote:

That is easy for you or i but i don't think my mom would know how to
do that. Are you actually suggesting it is an ok user experience for
users to *ever* have reconstruct URIs?

I'm saying it's a bad user experience NOT to allow URI construction, especially for sites that a person uses frequently.  

Your mom is part of a shrinking demographic; the growing demographic doesn't have the issues with technology.

The effort to hide certain technical concepts from users is counter productive; it simply makes them ignorant of important concepts.  The concept of a URL transcends technology, it's is a universal identifier for resources and efforts to hide them vs. present them in a manner people become familiar with do more harm than good, IMO.

Remember most users are familiar with file extension from their local computers.  Most users understand that if they see whitepaper.pdf on their local computer that it is an PDF document; it's similarly familiar for them to download a PDF from http://example.com/whitepaper.pdf whereas downloading a PDF from http://example.com/whitepaper is just weird and confusing.

User are resilient. Their strategy for recovering from a failure of
this sort is usually to find a competitor whose URIs just work.

Assuming there is a competitor sitting there with an alternate based on the URL problem.  But your comment is an abstract strawman because you haven't explained how it your vision would improve things for the user. How is a user better off with conneg?  Instead of potentially know how to hack to URL with conneg they are simple given no option to get what they want.

Users have absolutely zero interest in this sort of thing.

Funny, I've talked with many users whose opinion would invalidate that statement.  SOME users have no interest in this, but OTHERS care about it greatly.

They just give the uri to a user agent (usually by clicking it) and the user
agent handles all that administrivia. The application GETing (etc) the
uri is the thing that should decide what format is needed, not the
user.

Are you saying that if as a user I want an HTML page that the application should be in control so it could give me a PDF instead?  Not if I can help it.

The list of all known file extensions is long a growing. Do you really
want to maintain that regexp?

You don't need the full list, you only need the list the client supports.  I doubt most clients are going to process more than a few media types, and for those that process a lot, they can use a switch instead.   Looking at the IANA list of media types[1] we're still talking about an insignificant number of bytes to download once and cache. Smaller than all but the smallest of GIF images.

[1] http://www.iana.org/assignments/media-types/index.html

And that pattern ignores the servers the
don't use file extension but use a `?format=...` query string, or that
use an api version id in the uri, etc.

Since I argue against the ?format= pattern too, I ignored on purpose.

The alternate link rel is standardized[1], btw. If you are going to do
hyper media driven content negotiation please, please, please use
rel=alternate link headers. At least then i will not have to
find/write a parser for all the different formats that could be
returned and understand some custom reinvention of the wheel.

Did you miss the part where I said " mine were just quick and unvetted examples of URL extensions mapping to media types?"  The key argument was for the value of extensions to denote media types, now how they get mapped. But it we are going to create a standard why not standardize mapping?

Even it you do use standard alternate links the client is going to be
more complicated. It is also going to have to, potentially at least,
make more requests against the server.

Sounds like are you arguing for what makes it easier for how you currently develop, and easier not for users nor for developers in general.

I've already explained why IMO it's better for the user. For developers writing a simple API client for a simple (and mostly) RESTful web service it's much easier to specify JSON in the extension than to have to pass a header. The following code in PHP can give a products list, but to pass headers I have to do pull out ~20 lines of CURL:

$products = file_get_contents( "http://www.example.com/products.json" );

If OTOH you are building an advanced fully hypermedia driven API then you've got a lot more skill than many developers and you've got the skill to do a little extra work to get the information you need. And since you know what you are doing, you can use HTTP caching to minimize requests.

Having exactly one representation for every resource may be simple but
it is far from elegant. It increases the number of requests clients
must make (clients must sometimes remake requests based on alternate
links).

Rather than advocate against having extensions designating media types, which some people prefer more than conneg, why not just advocate both styles for an API:

http://www.example.com/products.{type} => returns {media type} mapped to {extension}. 

http://www.example.com/products  => Uses content type negotiation for HTTP header.

It makes identity harder to establish (is
"api.example.com/order/42.xml" the same order as
"example.com/order/42" or not?). 

Why does that matter?  If it matters, provide a link to the canonical resource in each representation.

It makes it harder to evolve clients
to new formats (if a client wants to move from some custom xml to
json-ld and it has a bunch of urls for xml representations bookmarked,
it must transform those URIs or support both xml and json-ld).

What I already described makes this migration trivial. I don't see your issue.  Note that the server must define the rules, but once the server has define the rules this is really not hard to deal with.

It is also harder on the server side because you have to be very disciplined
about providing the alternate links everywhere or clients are
potentially dead in the water.

Why?  The root of the API url for the API can provide the info. Call it once, cache it and you are done.

This feels like it's a case of not wanting URL extensions to be used.  If you don't want to use them you can always find reasons to advocate against them. Me, I want APIs to use extensions so it simplifies API use OTOH I don't have an issue if API also off conneg URIs too. Thus far I don't see significant issues with using extensions to map to media types; nothing you've explained appears to be an issue to me.  Maybe I'm still missing something, I'll admit that, but thus far I don't see any real downside, only upside.

-Mike

[1]: http://tools.ietf.org/html/rfc5988#section-6.2.2

Peter
barelyenough.org

[Steve Klabnik]

I think you seriously over-estimate 'normal users.' not in intelligence, but in the amount that they care about any of this. Remember the 'Facebook login' debacle? Users barely understand (read: care about) urls in the first place, let alone 'file extensions' (see: Mac os x popularity) or other unimportant details.

[Mike Schinkel]

On Sep 13, 2012, at 11:07 PM, Steve Klabnik <st...@steveklabnik.com> wrote:
> I think you seriously over-estimate 'normal users.' not in intelligence, but in the amount that they care about any of this. Remember the 'Facebook login' debacle? Users barely understand (read: care about) urls in the first place, let alone 'file extensions' (see: Mac os x popularity) or other unimportant details.

I don't overestimate, I just don't buy into the "lowest common denominator" approach to technology being the best.

And yes Mac OS X is a great example, I type this on a MacBook while upset that OS X keeps dumbing things down and justifies by saying "users are too dumb for that." I can go into detailed specifics about how the Mac makes things that could be easy hard, but this isn't the list for that.

-Mike

[Gabor]

Mike, all,

I think Apple's approach ("users are too dumb for that") is just being confirmed on a day-by-day basis - just look at their stock. It's not an accident why people buy their products in favor of others' - they just don't care about the technical details.

I had been working on Nokia products for a long-time before Apple entered the mobile market. What Nokia made well was that they had kept a lot of technical details organized, but not hidden away from users. This organizational thinking made them better than competitors (among others). Then entered Apple on mobile market and changed everything - users realized they didn't have to be confused by technology, rather technology must be adjusted to their needs. It might be an approach you don't prefer, but this is reality - people like simple things.

Now it's another question that we're talking about APIs here whose "customers" are programmers. And it's the apps that are used by real end-users. I must admit that I just realized that I'm not as knowledgeable about this topic as you. That's not a problem as I'm keen to learn. My point was to use already proven techniques (sometimes they're real standards, sometimes they're just "de facto") as long as they serve the purpose. Let's not forget about the fact, either, that <link rel=...> is an HTML stuff and we're talking about Web Services that might potentially have nothing to do with HTML (or there are a lot of steps in between a WS and some web representation).

Finally, going back to the original topic: I just don't think we must serve API consumers (i.e. programmers, apps) at the price that it causes us productivity issues. The API as a product, they are our most important customers, but let's not shoot ourselves in the foot.

Gabor

[Mike Schinkel]

On Sep 14, 2012, at 2:52 AM, Gabor <gabor....@gmail.com> wrote:
> I think Apple's approach ("users are too dumb for that") is just being confirmed on a day-by-day basis - just look at their stock. It's not an accident why people buy their products in favor of others' - they just don't care about the technical details.

Are you really going to use this forum to present a paean to Apple, to profess your belief that if Apple does it then it must be right?

Yes Apple has been very successful with a segment of the population, and simplicity has been one of it's hallmarks. But the operative word here is *segment*, not the entire population. Using your logic we should enshrine religion into law since ~86% of the world's population are religious. But that ignores those who are not, and it ignores the wide variance in those who are.

And just because Apple has been successful with a simplicity strategy doesn't mean that everything they've done is optimal, and certainly not optimal for all people. To say if Apple does it is the right approach is simply being a cargo-cultist as a participant in the modern world.

Further it ignores the fact that in many ways Apple is NOT simple, ever open a terminal window? I know you have. But without the power available in the terminal window all the power users I know would be using different computers. Most of the power users I know are programmers who love Unix, the anthesis of simple, but they are also computer users of apps they didn't program. With the current movement to learn programming by many people, including NYC Mayor Bloomberg it's not unreasonable to expect that a significant segment of technical savvy users will continue to exist and for most of the time on their computer, they are still users.

Coming back to the topic at hand, we are talking about whether extensions in URLs to indicate representation type will *harm* users or not, which I think is absurd. Are you too asserting that it harms users? My position is that it significantly *helps* users who understand them and the rest simply ignore it; no harm, no foul.

The problem with the "simplicity in all things" meme is that it ignores that when users actively use something they have they propensity to develop expertise and simplicity then holds them back. In my "Human Factors in Software Development" course in college the professor argued that the problem with Apple's approach is that it does not offer a "transitional user interface", which is great for beginners and casual users but greatly retards the abilities of those who are experienced.

Yes Apple has addressed the tech world's otherwise error in exposing too much complexity, but that doesn't mean their approach in ignoring the needs of experienced users is perfection either. Moderation in all things.

> Finally, going back to the original topic: I just don't think we must serve API consumers (i.e. programmers, apps) at the price that it causes us productivity issues. The API as a product, they are our most important customers, but let's not shoot ourselves in the foot.

Can you give me one concrete example of things we've been discussing (in this thread) that cause "us" productivity issues?

-Mike

[Peter Williams]

Just to be clear, i think content type specific URIs are a fine tool.
They are great when you have a resource whose flavor of representation
is an important characteristic (`/whitepaper.pdf` for example). They
are great for debugging. We should use them when they are useful.

Just because content type specific URIs are sometimes useful does not
mean we should use them for everything. Using the accept header field
for content negotiation results in a better experience for developers
(both server side and client side) and users because it results in a
combined system that is -- on the whole -- simpler, requires less
manual human intervention and is more resilient to change.

Peter
barelyenough.org

[Mike Kelly]

fwiw:

http://www.alvestrand.no/pipermail/ietf-types/2006-April/001707.html

> --
> You received this message because you are subscribed to the Google Groups "API Craft" group.
> To unsubscribe from this group, send email to api-craft+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/api-craft?hl=en.
>
>

[Gabor]

Dear Mike,

I don't know how much time it took for you to write your lengthy response, but I'm not sure it was worth it. You put so much passion into misinterpreting what I wrote that I don't think it'd be worth continuing the flame war you just started. Yes, you, because I didn't mean what you wrote. Anyway, I stop here. Thanks for the otherwise useful comments you made so far.

Gabor

[Peter Williams]

On Fri, Sep 14, 2012 at 9:06 AM, Mike Kelly <mikeke...@gmail.com> wrote:
> fwiw:
>
> http://www.alvestrand.no/pipermail/ietf-types/2006-April/001707.html

I disagree with his assertions.

First, content negotiation works pretty well for browsers. They
declare the relatively small number of mime types they understand
natively and the add */* with a low q value and dynamically look up
handlers for other types. This means they can give the user the best
experience when HTML, etc is available and a pretty good experience
when it is not.

Second, it works even better for most automata. Automata rarely
support a large number of media types so being able to inform the
server of exactly what is needed it extra important.

I have been using header based content negotiation in APIs for a long
time (>5 years) and it works great. There is obviously not a consensus
on whether header based content negotiation is the best approach but
let's not pretend that it doesn't work.

Peter
barelyenough.org

[Mike Schinkel]

On Sep 14, 2012, at 11:24 AM, Peter Williams <pe...@barelyenough.org> wrote:
> First, content negotiation works pretty well for browsers. They
> declare the relatively small number of mime types they understand
> natively and the add */* with a low q value and dynamically look up
> handlers for other types. This means they can give the user the best
> experience when HTML, etc is available and a pretty good experience
> when it is not.

When in practice is that actually used? Can you point to a URL that demonstrates it?

> Second, it works even better for most automata. Automata rarely
> support a large number of media types so being able to inform the
> server of exactly what is needed it extra important.

If the client knows that it needs a media type why can't it just pass that knowledge on the URL?

> I have been using header based content negotiation in APIs for a long
> time (>5 years) and it works great. There is obviously not a consensus
> on whether header based content negotiation is the best approach but
> let's not pretend that it doesn't work.

Can you explain a use-case where it has worked great that it was a better solution than content-type specific URLs?

-Mike

[Mike Schinkel]

On Sep 14, 2012, at 11:00 AM, Peter Williams <pe...@barelyenough.org> wrote:

Just because content type specific URIs are sometimes useful does not
mean we should use them for everything. Using the accept header field
for content negotiation results in a better experience for developers
(both server side and client side) 

Not clear if you are saying that URLs should only support content negotiation, or if you agree supporting both extensions and conneg make sense?

and users because it results in a
combined system that is -- on the whole -- simpler, requires less
manual human intervention and is more resilient to change.

As a user I find content negotiation very frustrating.  If I want to read the French version of a document I don't want my browser deciding that because my default language is English that I should only get to view English, for example.

-Mike

[Dan Schlossberg]

Great example mike.   If I want to see a page in Latin and the API allows me to do that by appending .latin that's great.  

On Sep 14, 2012, at 11:00 AM, Peter Williams <pe...@barelyenough.org> wrote:

Just because content type specific URIs are sometimes useful does not
mean we should use them for everything. Using the accept header field
for content negotiation results in a better experience for developers

(both server side and client side) <-- absurd, absolute, and without supporting example. Do u work for fox news?

I'm a developer.  I'd rather not think about headers at all.  

Not clear if you are saying that URLs should only support content negotiation, or if you agree supporting both extensions and conneg make sense?

and users because it results in a
combined system that is -- on the whole -- simpler, requires less

manual human intervention and is more resilient to change. <-- again, absolutist, unsupported claim. 

Huh? this is an API group.  The humans here are developers not app users. App users don't know API from dpi. And they certainly aren't going to modify a header. 

For the developer.   Human interaction is what we do to learn the API so we can automate it. .  The easier the api is to interact with, the better.  For a simple get, there is nothing easier than manipulating a URL that is already exposed and editable in an application I already have open.  If there is something easier, please tell us! 

As a user I find content negotiation very frustrating.  If I want to read the French version of a document I don't want my browser deciding that because my default language is English that I should only get to view English, for example.

-Mike

--

[Andrei Neculau]

Wow, this thread touched upon a broad range of topics, and I blame the question - spot on.

#1 Content-Negotiation

As with any negotiation, it may fail, and the one giving the answer is the server. You request A, but the server answers B. Perfectly fine on a protocol level, just as fine as server saying 406 Not Acceptable.

This becomes more "natural" or obvious, if you think of Accept-Language - client says it accepts Swedish, but the server only has the info in English, and thus has to decide whether to shut up or say something in English.

http://tools.ietf.org/html/draft-ietf-httpbis-p3-payload-18#section-6.1 
A request without any Accept header field implies that the user agent will accept any media type in response. If an Accept header field is present in a request and none of the available representations for the response have a media type that is listed as acceptable, the origin server MAY either honor the Accept header field by sending a 406 (Not Acceptable) response or disregard the Accept header field by treating the response as if it is not subject to content negotiation.

#1a http://www.alvestrand.no/pipermail/ietf-types/2006-April/001707.html

Content-Negotiation failed because browsers became too complex, accept a large variety of media-types and because the server can do UserAgent spoofing?! Just saying...

#1b "If I want to read the French version of a document I don't want my browser deciding that because my default language is English that I should only get to view English, for example."

That can surely be considered a missing browser feature - not being able to select and maybe even save the preferred language for a domain, or even a path. Other than that, it makes no case for overloading the meaning of an _opaque_ URI.

#2 the alternate link relation

"my.json" having a link with rel="alternate" towards "my.xml" does not mean anything, other than: this resource has an alternate URI. It would be the role of a type attribute to hint at the content-type that "my.xml" provides.

http://www.w3.org/TR/html5/links.html#rel-alternate 

The alternate keyword is used with the type attribute

#3 sharing links

Sharing a link gives absolutely no guarantee that the recipient will retrieve the exact same message - both in terms of content and format - as the sender. It's a false premise to assume so. I go to "example.com" from Sweden (with Accept-Language sv-se), I see it in Swedish, you visit it from the US (en-us), you see it in English. You give me a link from Chrome, while you see the resource it points to as text/html _representation_, but that link in my Lynx is shown via the resource's text/plain representation.

So, if you think that if you point me to "my.json" and we both must see a JSON structure, you're wrong. That's a fact that happens for a file-system, not for an API.

#4 an API is not for humans

Software does not care if you put "json" as variable A (URI) or variable B (Accept header). But software does care about how to treat variable A (known format, opaque meaning) and to treat variable B (known format, equal to the list of media-types that it can process, etc).

It's not more complex to say "please" when you ask for something. It's indeed more energy-consuming, time-consuming, but it has great benefits. So use the Accept headers, and not the URI, please!

Now it's another question that we're talking about APIs here whose "customers" are programmers. And it's the apps that are used by real end-users.

+1 Gabor

Now back to the primary topic.

URIs are opaque, both for the client and for the server. Yes, URIs have a protocol, a host, a path, a query, so they have a format, but they are still opaque. At most, you (server) look at the path in order to decide which "controller" to take the HTTP request.

If your controller is file-system based, then there is only one controller, which takes the whole path, applies that to the root folder, and returns whatever content is at the other end. That's why you need the extension, not because you instruct the server to give file.png as PNG. You only instructed the server to give you file.png, and your browser's Accept header said 'text/html, */*'. file.png can very well be a JPG file with the wrong extension, but the server doesn't know any better than to look at the file's extension and mapping it to a content-type (MIME), and that is why you get a response of content-type=image/png.

Same for using the query. How is "format=(xml,json)" a query to /cars/123 ? http://dictionary.reference.com/browse/query ? I understand to query a collection, or a service resource, to search, to filter, to paginate, but to request a format?

Having said that, a browser is a browser. A browser is not an API debugging environment. Can you just open a new tab now and do a POST? No. Can you open a new tab a send a request payload? No. Can you open a new tab and set headers of a future request? No. Obviously you can with extensions, but not with the browser alone.

If your API is actually a website, then you can look at the browser as a debugging environment - you debug HTML, JS, CSS, etc.

Bottom-line, design your API as best as you can by making use of the bare minimum: the HTTP protocol, and possibly the REST style. Don't design because it's easy for the browser to X, or for curl to Y, or, or.

PS: scratch the above fully, if your API is made up of 10 resources, won't change so much over time, etc, etc. Just stating the obvious: if you're small, if you just need to move from your home to the supermarket, if you think your product is rather ephemeral.. don't try designing a BMW - a carriage works just as well. But then let's not argue that a carriage is as good as a BMW, or that it defines or should define the car industry, just because it does the job, or because it was easier to build and debug (understand, explain).

[Mike Schinkel]

On Sep 20, 2012, at 5:12 PM, Andrei Neculau <andrei....@gmail.com> wrote:

#3 sharing links

Sharing a link gives absolutely no guarantee that the recipient will retrieve the exact same message - both in terms of content and format - as the sender. It's a false premise to assume so. I go to "example.com" from Sweden (with Accept-Language sv-se), I see it in Swedish, you visit it from the US (en-us), you see it in English.

Without re-reading the specs in detail, I'm going to assume you are correct as per spec.  

But, with caveats, that doesn't mean it is generally a good idea to serve something different when the same URL is requested at different times or by similar clients on different machines.  I'm not talking about specs but instead user-experience and what might arguably be considered a best practice that goes beyond what the specs require.  If I visit example.com and it's in English, if I send a link to a friend I would be surprised if he viewed it in Swedish. 

Anecdotally I've noticed that many sites that want to provide multi-language content do so by seeing the request and either asking what language the user wants, or guessing from the headers and/or lookup of the location of the attached IP address and then they redirect to an appropriate URL, i.e. http://example.com/en-us/ or http://example.com/sv-se/.  If I then send the link  http://example.com/en-us/a-great-story/ to a friend it would be particularly bizarre is the friend loaded it and found it in Sweden.

Here's a post published on this very subject:  

http://h3h.net/technology/designing-urls-for-multilingual-websites

In summary, I think best practices should be based on the principle of least surprise for the targeted user; if it would confuse a reasonable user then it's a bad idea; if it would make reasonable sense to a user it would be a good idea.  URLs are identifiers for resources and when dereferenced return representations; if those reputations are not what the targeted user would expect, then taking the liberties that the specs allows but that violates Jakob's Law of User Experience is just a bad idea. IMO. 

You give me a link from Chrome, while you see the resource it points to as text/html _representation_, but that link in my Lynx is shown via the resource's text/plain representation.

I'm pretty sure a reasonable user would understand that a client that cannot present a representation of the form that another client can would not display the more complex representation.  So that example is effectively a red herring.

#4 an API is not for humans

Software does not care if you put "json" as variable A (URI) or variable B (Accept header). But software does care about how to treat variable A (known format, opaque meaning) and to treat variable B (known format, equal to the list of media-types that it can process, etc).

It's not more complex to say "please" when you ask for something. It's indeed more energy-consuming, time-consuming, but it has great benefits. So use the Accept headers, and not the URI, please!

I've heard this argument numerous times, but it ignores the perspective that software is written *by* humans and the opinions of some that being easier for human developers to work will make it easier to build functional and robust client software.  I do understand that reasonable people will differ on what they value so we will likely have to agree to disagree and just hope we don't respectively have to write clients for APIs designed by the other. :)

Now back to the primary topic.

URIs are opaque, both for the client and for the server. Yes, URIs have a protocol, a host, a path, a query, so they have a format, but they are still opaque. At most, you (server) look at the path in order to decide which "controller" to take the HTTP request.

What you are saying about URLs being opaque is only true from certain perspectives and without explanation can lead people to believe that URLs MUST be opaque in all cases, which is not true.  Yes, a client or an intermediary cannot make assumptions about a URL according to the HTTP spec, but a server CAN publish additional logic about it's URLs and if so a client CAN treat it's URLs as less than opaque given the published documentation. 

So if we publish an API and in the docs we write that a URL with a ".json" extension will return data of "application/json" media type and ".xml" will return "application/xml", then the URL extension is NOT opaque regarding extensions to any client that wants to use that information to assemble it's URLs. It might be out-of-ban information, yes, but it's still a valid approach (although not fully RESTful as per Fielding.) Again, we are talking reasonable conventions for the target user, i.e the API developer in this case.

It comes down to what the server API architect wants to do.  If they want to use extensions to indicate media type, they can (and I hope they do for any APIs I'll need to write clients for.)

Having said that, a browser is a browser. A browser is not an API debugging environment.

Correct, it's not.  But to the extent it can be, it makes a developer's day much nicer.  At least this developer thinks so.

PS: scratch the above fully, if your API is made up of 10 resources, won't change so much over time, etc, etc. Just stating the obvious: if you're small, if you just need to move from your home to the supermarket, if you think your product is rather ephemeral.. don't try designing a BMW - a carriage works just as well. But then let's not argue that a carriage is as good as a BMW, or that it defines or should define the car industry, just because it does the job, or because it was easier to build and debug (understand, explain).

To that I agree 100%.  

To create a complex hypermedia API is a significant undertaking in terms of cost and timeline. If the cost and time to market is worth it to the API owner (to build a BMW) and the API client (to finance and then maintain a BMW) then by all means that's how it should be done.  But if the API is simple, OR if the API is going to be programmed by people with small budgets and/or little programming skill then making it as easy as possible to interface with is, to me, a really good idea.

-Mike

[Javier Cervantes]

I'm not so good in this and just stating to get involved in hypermedia, but I don't think it is that complex and I think it is worth to have decoupled client.

I have a small budget and simple API and I rather use Hypermedia because in the end it will be less expensive. (As it will be easier to maintain)

Regards

Javier

---

‎"Ningún descubrimiento se haría ya si nos contentásemos con lo que sabemos"

Séneca

--
You received this message because you are subscribed to the Google Groups "API Craft" group.
To unsubscribe from this group, send email to api-craft+...@googlegroups.com.

Visit this group at http://groups.google.com/group/api-craft?hl=en.
 
 

[Matthew Bishop]

I love this thread because it reveals a couple of interesting things:

1. Some client developers don't pay much attention to the media type. They do care about the serialization format, and so far only three exist in common usage: XML, HTML, and JSON. It's safe to say that content-type renegotiation will not lead to a successful exchange; if my client expects JSON but receives XML, their app will break. technically it doesn't have to, but in reality the client will not go to the trouble of creating a different thunker to deal with the format change. Instead they will file a bug, or yell at the server dev on a group or abandon the API altogether.

2. API developers do not develop clients. At least not nearly enough. If they did, they would realize how hard it is to actually consume the errors and headers. They would consider Postel's Principal and offer multiple ways for the client to communicate their intent via accept headers or file extensions.

Matt

[Peter Williams]

On Sep 21, 2012 10:09 AM, "Matthew Bishop" <ma...@thebishops.org> wrote:
>
> 2. API developers do not develop clients.

You are letting your prejudices show.

> At least
> not nearly enough. If they did, they would realize
> how hard it is to actually consume the errors and
> headers.

Ime, it is just false that client tool chains make it difficult to use headers. If the tools you use make it hard to do basic things like use headers then I suggest improving or replacing them. That level of incapability is no longer the norm.

> They would consider Postel's Principal and offer
> multiple ways for the client to communicate their
> intent via accept headers or file extensions.

Postel's principal does not call for documenting, publishing, and forever supporting all the weird, but suboptimal, ways one might, through the grace and skill of the api programmer, successfully use an api. Rather it calls for doing the best you can with what you get, regardless of whether you are on the client or server side.

Peter
barelyenough.org