Media Type Design - Should the media subtype describe the resource?

paulkmoore

unread,

Mar 21, 2012, 4:33:02 PM3/21/12

to hyperme...@googlegroups.com

Hi all,

Background

First posting here, so first a little background...

I am in the process of building a hypermedia based API adhering as closely as possible to the RESTful constraints. The API format is based on JSON [RFC4627], but also uses hypermedia controls in the form of Link Headers [RFC5988] and semantic serialisation of the Link Header for representation embedding purposes.

Question

My question is; in the design of a domain specific media type, what is the level of required specificity within the subtype field? Specifically, should the media type subtype include the type of resource for the particular representation?

Example & Options

To expound a little, and in the vernacular of the well loved Restbuck's example from 'Rest in Practise', consider that I have 'order' and 'receipt' resources. I could propose Media Types of the following forms:

Form 1: Specific Media Types

application/vnd.restbucks.order+json

application/vnd.restbucks.receipt+json

Form 2: Use of Media Type Parameters (à la ATOM Pub [RFC5023])

application/vnd.restbucks+json;type=order

application/vnd.restbucks+json;type=receipt

Form 3: Payload Inspection

application/vnd.restbucks+json

... and expect that the user-agent inspects the payload to determine the resource type.

Research

Investigation into HTTP/1.1 [RFC2616] largely defers media type concerns to the MIME RFCs.

From RFC2045, I note the following pursuant points:

"The purpose of the Content-Type field is to describe the data contained in the body fully enough that the receiving user agent can pick an appropriate agent or mechanism to present the data to the user, or otherwise deal with the data in an appropriate manner. The value in this field is called a media type."

and,

"The Content-Type header field specifies the nature of the data in the body of an entity by giving media type and subtype identifiers, and by providing auxiliary information that may be required for certain media types."

I note from HTTPbis (draft) - 4.2. Representation Data:

"Content-Type specifies the media type of the underlying data, which defines both the data format and how that data SHOULD be processed by the recipient (within the scope of the request method semantics)"

which echoes the definition of RFC2045.

Further in HTTPbis, and within the context of Content-Type mismatch handling, there is the following:

"Implementers are encouraged to provide a means of disabling such "content sniffing" when it is used."

This is supported by a ticket raised (by Mark Nottingham) against the draft, which suggests that the use of 'sniffing' (payload inspection) is disallowed by the HTTP specification itself. The ticket somewhat implicates payload inspection as being undesirable in the general case.

Mark Masse refers to this inspection technique as a REST anti-pattern (here), specifically:

“This anti-pattern is a form of tunneling, that we might call "type tunneling" [sic] because it masks the content's actual "type" and reveals only its format.”

which resonates.

Roy Fielding's edicts echo this approach; here:

“A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types.”

and here:

“... media types are the only means we have for an HTTP sender to express the intent for processing a given message payload. While some people have claimed that recipients should sniff the data format for type information, the fact is that all data formats correspond to multiple media types. Sniffing a media type is therefore inherently impossible: at best, it can indicate when a data format does not match the indicated media type; at worst, it breaks correct configurations and creates security holes. In any case, sniffing cannot determine the sender’s intent.”

Status Quo

I have investigated many “RESTful” protocols to understand how others have solved the issue and the general case appears to be “use payload inspection”! Arrrghhh!

There are notable exceptions, including:

Use of the "type" parameter in ATOM pub (RFC5023 - 12. The Atom Format Type Parameter),
Use of specific types in Oracle's Cloud API
Use of specific types in RunKeeper's API

Current View

My current view is that whilst the status quo is to use the payload inspection technique, this is somewhat unRESTful. I'm inclined to conclude that “Form 1: Specific Media Types” is the way to go.

Question (again)

My question is; in the design of a domain specific media type, what is the level of required specificity within the subtype field? Specifically, should the media type subtype include the type of resource for the particular representation?

Many thanks for your time, and considered responses

Paul

Mike Kelly

unread,

Mar 21, 2012, 5:56:45 PM3/21/12

to hyperme...@googlegroups.com

Hi Paul,

There is a 4th form which is similar to 3 in that the media type
identifier is entirely untyped but, rather than inspection, clients
determine the resource type from their preceding context i.e.
inferring the type against the link relation which lead to the
resource's discovery.

So if the client follows <link rel="coffee-menu" href="..." /> then it
can interpret the response as a representation of a coffee menu. This
is distinct from Form 3 as no 'payload inspection' is having to take
place.

If you structure your application this way, there is actually no need
to use a domain specific media type at all. Instead you can pick a
general-purpose linking media type and govern and express your domain
specifics via set of link relations instead.

Cheers,
Mike

mca

unread,

Mar 21, 2012, 6:32:03 PM3/21/12

to hyperme...@googlegroups.com

Paul:

good to see you here!

ok, a diff question altogether....

why do you need this "type of resource" identifier at all? what is it
used for on the client?

mca
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

> --
> You received this message because you are subscribed to the Google Groups
> "Hypermedia Web" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/hypermedia-web/-/KTjnUOo1WHsJ.
> To post to this group, send email to hyperme...@googlegroups.com.
> To unsubscribe from this group, send email to
> hypermedia-we...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/hypermedia-web?hl=en.

Paul Moore

unread,

Mar 22, 2012, 9:13:42 AM3/22/12

to hyperme...@googlegroups.com

Mike

Great to be here :-)

The "need" point is a good question. At the moment I use the 'rel' attribute to infer the 'nature of the resource'. The client is a user-agent that synchronises resources with server, and has an entirely separate (human) User Interface. The user-agent is therefore from the automaton category, rather than analogous to a browser per se. As an aside the client dereferences relevant controls to converge on a consistent object graph on the client.

Whilst this is OK - and maybe even 'good', I'm concerned that a user who entered the API outside of a hypermedia control (i.e. followed a 'plain' link from an email) has no context on which to infer the 'nature of the resource' (see in part my discussion with Ian here) - hence the consideration of resource specific media types. Indeed, part of the MIME specification is to ensure that payload is adequately described by the media type, as part of the 'packaging'.

Revisiting Roy's dissertation,

From Section 5.2.1.1; "A representation is a sequence of bytes, plus representation metadata to describe those bytes" and from Table 5-1; "representation metadata" includes "media type" and "last-modified time".

So I think if I don't "describe those bytes" I'll be breaking the 'self-descriptive messages' constraint.

Conversely, in Section 6.3.4.5, Roy states "Unfortunately, MIME and HTTP have very different goals, and the syntax is only designed for MIME’s goals.". He further describes some of the packaging aspects of MIME which are, by inference, part of the 'different goals'. Unfortunately HTTP/1.1 [RFC2616] does little to clarify the situation, largely deferring to MIME.

From Roy @ Untangled:

"A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API)." and "[Failure here implies that out-of-band information is driving interaction instead of hypertext.]"

Which hints that using (specific) media types is 'bad' because the only thing a user-agent should be doing is following hypermedia controls, not inferring resource type information from the media type. So, if I use specific media types I break the 'hypermedia as the engine of application state' constraint.

Two choices, both which appear to conflict.

To go back to the "need" point - I "need" to not have redesign / rebuild my API again! I would like to finalise my conceptual model before I spend any more time re-factoring code, and some of these concerns are fairly fundamental.

Paul

Darrel Miller

unread,

Mar 22, 2012, 9:29:31 AM3/22/12

to hyperme...@googlegroups.com

Paul,

It's a great question. Unfortunately I don't think you are going to get a clear answer. From my perspective, REST is about decoupling client and server. The more application specific your media type/sub-type is the more coupling you are creating between client and server. I believe that ideally, the media type should contain just enough semantics to allow the client to do its job but no more. How much semantics are required depends on how much intelligence you want to put in your client. I create dumb clients, so my media types are pretty generic. I do think that creating media types down to the level of types/business entities is too specific. I prefer to look for more horizontal concepts like lists, questions, statuses, policies, permissions, transactions, party, person, etc. These kinds of generic concepts can be combined in application specific contexts to create real business workflows.

As general guidance, I would try and create as generic a media type/sub-type as possible without requiring your client to have out of band information, how generic is generic, well that depends on the scope of your application.

Darrel

Markus Lanthaler

unread,

Mar 22, 2012, 10:25:12 AM3/22/12

to hyperme...@googlegroups.com

Hi Paul,

I think you are partly confused due to the term “content sniffing”. A media
type is used to be able to select a processor capable of processing a
representation. It is OK to have enough semantics **within** the
representation to distinguish between different entities.

IMHO you don't need to create a new media type for every single entity as
long as they have the same serialization format and thus processing model -
I would actually discourage such an approach. So, taking a JSON-based media
type as example, it would be completely fine to have a "type" property at
the top-level such as in:

{
"type": "order",
...
}

or

{
"type": "receipt",
...
}

to distinguish between the different entities.

Hope this helps,
Markus

--
Markus Lanthaler
@markuslanthaler

----------------
From: hyperme...@googlegroups.com
[mailto:hyperme...@googlegroups.com] On Behalf Of Paul Moore
Sent: Thursday, March 22, 2012 9:14 PM
To: hyperme...@googlegroups.com
Subject: Re: Media Type Design - Should the media subtype describe the
resource?

Nick Gall

unread,

Mar 22, 2012, 10:35:26 AM3/22/12

to hyperme...@googlegroups.com

On Thu, Mar 22, 2012 at 9:13 AM, Paul Moore <paul.k...@lineone.net> wrote:

The "need" point is a good question. At the moment I use the 'rel' attribute to infer the 'nature of the resource'. The client is a user-agent that synchronises resources with server, and has an entirely separate (human) User Interface. The user-agent is therefore from the automaton category, rather than analogous to a browser per se. As an aside the client dereferences relevant controls to converge on a consistent object graph on the client.

Whilst this is OK - and maybe even 'good', I'm concerned that a user who entered the API outside of a hypermedia control (i.e. followed a 'plain' link from an email) has no context on which to infer the 'nature of the resource' (see in part my discussion with Ian here) - hence the consideration of resource specific media types. Indeed, part of the MIME specification is to ensure that payload is adequately described by the media type, as part of the 'packaging'.

...

So I think if I don't "describe those bytes" I'll be breaking the 'self-descriptive messages' constraint.

I like where Mike Kelly is going. Keep media types as generic as possible and use relation names and markup for domain specific "typing", which is compatible with the advice from Roy that you quote.

One way to do this is to use the "category" element offered by the ATOM Publishing Protocol. See http://tools.ietf.org/html/rfc4287#section-4.2.2 . I believe this approach is used by Microsoft's OData Protocol. See http://www.odata.org/developers/protocols/atom-format#RepresentingEntries . For example:


<entry xml:base="http://services.odata.org/(S(S_ODATA_RO))/OData/OData.svc/"
    xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
    xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
    m:etag="W/"0"" xmlns="http://www.w3.org/2005/Atom" Xmlns:c="http://customUri">
  <id>http://services.odata.org/(S(S_ODATA_RO))/OData/OData.svc/Suppliers(0)</id>
  <title type="text">Exotic Liquids</title>
  <updated>2010-03-11T22:47:21Z</updated>
  <author>
    <name />
  </author>
  <link rel="edit" title="Supplier" href="Suppliers(0)" />
  <link 
      rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Products"
      type="application/atom+xml;type=feed" title="Products"
      href="Suppliers(0)/Products" />
  <category term="ODataDemo.Supplier"
      scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
  <c:Road>NE 228th</c:Road>
  <content type="application/xml">
    <m:properties>
      <d:ID m:type="Edm.Int32">0</d:ID>
      <d:Name>Exotic Liquids</d:Name>
      <d:Concurrency m:type="Edm.Int32">0</d:Concurrency>
      <d:Address m:type="ODataDemo.Address">
        <d:Street>NE 228th</d:Street>
        <d:City>Sammamish</d:City>
        <d:State>WA</d:State>
        <d:ZipCode>98074</d:ZipCode>
        <d:Country>USA</d:Country>
      </d:Address>
    </m:properties>
  </content>
</entry>

BTW, OData has a JSON serialization as well, so you could use OData right out of the box.

-- Nick

mca

unread,

Mar 22, 2012, 11:10:11 AM3/22/12

to hyperme...@googlegroups.com

First, using media type decorators (i.,e. sub-types
"application/vnd.pkmoore+json;type=order") is essentially minting a
new media type. IOW, the client must "code for" the media type as it's
primary focus. In my experience, this leads to "teaching" dozens of
processing models to a client just to handle a single problem domain
(how many "sub-types" would there be in an accounting system?)

OTOH, I like Markus' POV. if the recipient will need some metadata
about the data, include that in the representation.

HTML:
<div class="order">
...
</div>

XML
<root type="order">
..
</root>
or just:
<order>
...
</order>

JSON
{
type: "order",
{...}
}
or just
{ order : {...}}

In the above examples, you write a single media type processor
(application/vnd.pkmoore+json) and let the local client handle any
specifics about the payload when (and if) that is needed.

Also, as Darrel points out, you'll not get a clear cut answer on this
topic as this issue is about _design_, not standards. Hopefully you'll
get enough cogent feedback to help you make an informed decision that
makes sense for your users/devs.

mca
http://amundsen.com/blog/
http://twitter.com@mamund
http://mamund.com/foaf.rdf#me

paulkmoore

unread,

Mar 22, 2012, 11:39:21 AM3/22/12

to hyperme...@googlegroups.com

Mike

Thanks for this. I was actually aware (and had been using) this 4th form, but in discussions elsewhere I discounted it. I want to revisit that decision before replying here with a (hopefully) considered response.

Thanks

Paul

Jørn Wildt

unread,

Mar 22, 2012, 11:42:21 AM3/22/12

to hyperme...@googlegroups.com

I have been searching for an answer for this question too for some time now. Just to give you an impression of how much debate it causes, you can try to go through this thread: http://tech.groups.yahoo.com/group/rest-discuss/message/18183?var=1&l=1 (and it goes on in other threads from that).

Just like Ian states in https://groups.google.com/d/msg/restinpractice/lvmxlyHaT-k/7pTqswTYlNsJ, I also tend to change my mind upon this question over time. So, here is my current view ...

A media type describes how to "process" a resource. By "processing" I think of how the client parses the content (is it XML/HTML/CSV/JPEG/PDF/other), and how it interacts with the resource in terms of hyper-media controls (HTML has anchors, forms and more, ATOM has links and so on).

Some media-types have no hyper-media controls - like for instance an image or a spreadsheet. So they say nothing about how you can interact with the resource. Some media-types have variations that contains hyper-media controls - like for instance application/atom+xml which has links embedded. And some media-types has lots of hyper-media controls - like for instance HTML with its anchors, links, images, forms and so on.

So far none of the mentioned media-types carry any application semantics (or "type" information) like "this is a purchase order" or a "Person". But those media-types that has hyper-media controls tells you how you can find other resources (links) or change data (forms). This is the "processing model" of the media-type: how to parse, navigate and interact with the resources.

Here is a few examples of media-types and their processing model (with increasingly more hyper-media controls):

text/plain: its a text file.

text/csv: its a text file and you can read tabular data from it.

image/jpeg: its an image.

application/xml: its XML, no more, no less.

application/json: its JSON, no more, no less.

application/pdf: its displayable as PDF and has links to other resources, you can navigate it.

application/atom+xml: its a list, it contains links to other resources, you can navigate it.

text/html: its HTML and you can do lots of wonderful stuff with it.

If the agent interacting with the resource was a human being then the story could stop here. A human would be able to reason about the content and understand the application level semantics by looking at, for instance, the rendered HTML. But a computer cannot reason about fluffy stuff like HTML (well, actually it can, I will get back to that). So, somehow we have to add more semantics to the content.

What is needed is something that tells us "what type of data does this resource represent?". We already know how to interact with it (assuming it is a hyper-media enabled media-type), but we do not know the "vocabulary" used. For me a vocabulary is a description of which properties I can find in the payload - and what they mean for my application domain. The processing model already tells us how to parse the data, and the vocabulary adds meaning to the parsed data.

The question is now: should we put the vocabulary reference in the media-type (like "vnd.person" in application/vnd.person+xml) or somewhere else?

If media-types are about "processing model" (in the above meaning) then we should only mint new media-types if we change the processing model - right? We can use this as a guideline: does the data type introduce new hyper-media controls or other stuff that changes the processing model?

So, to answer the questing about "vnd.person", we ask "Does a person introduce new processing instructions?". I would say, no, it does not, so we should not mint a new media-type for a person.

Lets try with a different beast: "Does ATOM introduce new processing instructions?". Yes, it adds links (and a "list" structure, but I don't really count that as a processing instruction), so it is reasonable to introduce the application/atom+xml media-type.

And here is one more sanity test: "Does JPEG images introduce new processing instructions?". Well, it does not add any hyper-media controls, but it certainly introduces new parsing instructions. So it also makes sense to introduce image/jpeg.

So we should not introduce application/vnd.person+xml, but how do we then tell the user agent how to understand the content after it has been parsed? Unfortunately I haven't yet seen anything that people can agree on ...

But let me illustrate the problem from a slightly different angle: what if the media-type was text/csv (tabular data), or even application/x-www-form-urlencoded (key/value pairs) - or maybe even application/pdf? It is perfectly valid to represent a person using any of these media-types - and they all have good use-cases.

In XML we are used to slap something like "vnd.person" into the media-type - but there is no offical way to do that for PDF, CSV or URL-encoded. There is no "text/vnd.person+csv" or "application/vnd.person+pdf". So where should we put the indication of the vocabulary for those media-types?

In CVS we can let the first (top) left cell contain the name of the vocabulary, and then use that information to interpret the remaining columns. In XML we can let the namespace or root element specify the vocabulary - and PDF, nah, that's for human reading mostly :-) But there is no standard way to do it.

It would be nice though if there was some kind of standard HTTP header that could carry the vocabulary information. Maybe Eric Wilde's "profile link relation type" could be used as a generic solution (see http://dret.net/netdret/docs/draft-wilde-profile-link-00.html)?

For machine-to-machine (M2M) interaction it can be interesting to look into HAL (http://stateless.co/hal_specification.html) which introduces a simple XML format for reading and linking data. Unfortunately it does not (yet) specify anything with respect to changing data.

Even HTML is interesting from a M2M point of view: it has links and forms, but it lacks a standardized way of representing "type" information (like "persons name" and so on), and it also lacks the ability to specify other data modifying methods than POST (in forms). Still, you can get far with stuff like micro formats - and it is even human readable (and "interactable") at the same time.

Unfortunately I have not yet seen any generic format that includes all of the HTTP methods. So, for now you have to mint your own media-type (often an XML variation) in order to tell the client how to DELETE or PUT data. But don't do it for "Person", "Sales order" and so on.

You are also free to combine stuff: use HAL for reading data and embed links to resources with HTML forms for updating stuff - or to resources with your own media-type for updating resources.

/Jørn