Patterns for a RESTful SOA

46 views
Skip to first unread message

Ernest Prabhakar

unread,
Nov 6, 2008, 1:21:38 PM11/6/08
to restfu...@googlegroups.com
Hi all,

I just saw this presentation, and I thought it did a brilliant job of
describing RESTful concepts -- and implementation issues:

http://akamai.infoworld.com/weblog/stratdev/archives/Patterns.pdf

In particular, lists three things that Atom does which are really
useful for a services architecture:

> atom:id
> atom:updated
> atom:link

Something to think about for JSONPub (or whatever we're calling it
these days).

I've added it to the REST wiki:

http://microformats.org/wiki/rest#Resources

-- Ernie P.

Kris Zyp

unread,
Nov 12, 2008, 11:38:15 AM11/12/08
to restfu...@googlegroups.com
I wanted to propose another possible approach for defining a JSON
structure for RESTful interaction. I have previously suggested the
importance of unobtrusively allowing JSON to preserve to it's natural
structure within a RESTful architecture; alternate JSON media sub-types
and structures should certainly be usable within a JSON-base REST
architecture, as alternate media-types are a core aspect of REST. At the
same time, others have expressed the desire to be able to include
resource metadata within JSON data structures. Perhaps the best approach
would be rather than enforcing a certain structure for resource
metadata, allow JSON representations to define their own structure or
schema and ascribe their own meaning for the hierarchy of their
structure. A JSON Schema [1] definition of data could be used (which
also provides definitions of allowable data types, very useful for
modifying data) and augmented with resource metadata types that could be
assigned to different properties. Additionally, this would permit
servers to define their own properties to use for hyperlinks and
resource identification. I believe it is one of our goals that we should
make it easy as possible for existing implementations to become
interoperable without forcing them to create their own backwards
incompatibility issues. Essentially this approach would mean they we
would be creating a meta-specification for defining how to specify
RESTful JSON.

There are several advantages to this approach:
1. Maintain unobtrusive natural JSON representations. Servers can
describe the structure and meaning of their JSON in terms of how it maps
to REST concepts (URLs, content types, etc.), rather than being forced
to follow a certain structure. Virtually any JSON structure that
contains RESTful information can be understood by clients with modifying
the structure. Existing implementations can conform by providing a
schema (and pointing to it in the MIME parameter) without any
modification of their outputted data.
2. We can include metadata within JSON representations without needing
separate content types
3. JSON Schemas on their own are an invaluable resource for user agents
that may be modifying resources based on JSON representations. Schemas
can be used to user agents to understand what are valid values for
different properties and properly constrain and guide the user in
updating data. A schema can be central resource for guiding the user
agent in both property constraints as well understanding the REST
mappings of the properties.

The main disadvantage is that it increases the complexity of user
agents, and what they must understand (they can't hardcode the property
names for references/links for example).

There are a number of ways that one could specify the applicable schema
for a JSON representation, but like I had suggested with the "locator"
property, I believe a MIME type parameter is most appropriate since it
does not impose on the representation format. For example, an object
representation could look like:

GET /Project/proposal

Content-Type: application/project+json; schema="/projectSchema"
{
// metadata:
"type": "application/project+json",
"href": "proposal",
"id": "proposal",
// data:
"data": {
"projectOwner": {
// metadata:
"href": "/Person/kris",
"id":"kris",
// data:
"name":"Kris Zyp"
}
}
}

And we could define a schema:
Content-Type: application/schema+json;
schema="http://json-schema.org/schema"
{
"metadata": { # mappings of our properties to REST metadata
"contentTypeProperty":"type",
"hrefProperty":"href"
},
"envelopeFor": "data", # indicating that the value of the data is
the real representation
"properties":{
"projectOwner":{ # schema for projectOwner values
"properties:"{
"name":{"type":"string"} # this is a JSON Schema
property constraint (value must be a string)
}
}
}
}

This schema provides all the information necessary for a user agent to
understand the properties of the instance (/Project/proposal).

An example of a collection:
GET /Project/
Content-Type: application/myapp+json; schema="/projectCollectionSchema"
{
"lastModified":"some date",
"members":[
{
// metadata:
"type": "application/project+json",
"href": "proposal",
"id": "proposal",
....

]
}

And a schema for the collection:
GET /projectCollectionSchema
Content-Type: application/schema+json;
schema="http://json-schema.org/schema"
{
"metadata": { # mappings of our properties to REST metadata
"lastModifiedProperty":"lastModified"
},
"envelopeFor": "members", # indicating that the value of the members
is the "real" representation
"items":{ # definition of the items in the array/collection
# hyperlink to the schema for each project object
# using the hyperlink property defined at
http://json-schema.org/schema
"$ref":"projectSchema"
}
}

Additional definitions that could be provided in the schema (and not
shown in these examples) could be differenting between a URL property
that does provide a full object representation (locator) vs one that
points to another location to retrieve the data (href), and perhaps
includes a partial/summary representation. Also, a baseUrl could be
defined for subobjects to support leveraging existing identifier
properties in more complex URL schemes (like those used by Rails).

Thanks,
Kris

[1] http://json-schema.org and http://groups.google.com/group/json-schema

David Snopek

unread,
Nov 12, 2008, 2:05:57 PM11/12/08
to restfu...@googlegroups.com
Kris,

I think this is a great idea. It could allow several different JSON
media-types to interoperate without a user agent having to be
explicitly taught how to deal with each one.

Cheers,
David.

2008/11/12 Kris Zyp <kri...@gmail.com>:

--
Open Source Hacker and Language Learner
http://www.hackyourlife.org/

Robert Brewer

unread,
Nov 12, 2008, 2:49:46 PM11/12/08
to restfu...@googlegroups.com

Sounds like you're trying to turn JSON into XML.

> There are several advantages to this approach:
> 1. Maintain unobtrusive natural JSON representations. Servers can
> describe the structure and meaning of their JSON in terms of how it
> maps
> to REST concepts (URLs, content types, etc.), rather than being forced
> to follow a certain structure. Virtually any JSON structure that
> contains RESTful information can be understood by clients with
> modifying
> the structure. Existing implementations can conform by providing a
> schema (and pointing to it in the MIME parameter) without any
> modification of their outputted data.
> 2. We can include metadata within JSON representations without needing
> separate content types

Really? The existence of AtomPub would seem to argue against that. If
even XML needs separate content-types, why wouldn't JSON?

> ...
> Content-Type: application/project+json; schema="/projectSchema"
> ...
> Content-Type: application/schema+json;
> ...
> Content-Type: application/myapp+json;
schema="/projectCollectionSchema"

...and then you go and invent 3 new content-types. ;)

Also, http://tools.ietf.org/html/rfc4288#section-4.2 says:

In accordance with the rules specified in [RFC3023], media subtypes
that do not represent XML entities MUST NOT be given a name that ends
with the "+xml" suffix. More generally, "+suffix" constructs should
be used with care, given the possibility of conflicts with future
suffix definitions.

So unless you're gunning to be the author of those "future suffix
definitions" (and it sounds like you might be), be careful with
"application/*+json" types.

I'm headed in the other direction. I think we can have a spec like an
AtomPub without needing a "JSON media types" spec a la RFC 3023, and
without needing generic JSON schema specs. I'm about 90% done with a new
media-type along those lines which I'll try to share soon.


Robert Brewer
fuma...@aminus.org

Alexandros Marinos

unread,
Nov 12, 2008, 2:58:36 PM11/12/08
to restfu...@googlegroups.com
Hi kris,

This approach seems to be creating a meta-media type which I believe is a good approach as current media-types are quite hard to extend and also mix syntax and semantics. However, a similar proposal was going around in the REST cycles a while ago and it did get shot down. Some coverage can be found in stefan tilkov's blog:

http://www.innoq.com/blog/st/2008/02/decentralizing_media_types.html

http://www.innoq.com/blog/st/2008/02/media_types.html

http://www.innoq.com/blog/st/2008/02/more_about_decentralized_media.html

Perhaps the RESTful JSON forum is a better place to evolve such a proposal?

Kris Zyp

unread,
Nov 12, 2008, 3:07:03 PM11/12/08
to restfu...@googlegroups.com

>> the structure. Existing implementations can conform by providing a
>> schema (and pointing to it in the MIME parameter) without any
>> modification of their outputted data.
>> 2. We can include metadata within JSON representations without needing
>> separate content types
>>
>
> Really? The existence of AtomPub would seem to argue against that. If
> even XML needs separate content-types, why wouldn't JSON?
>
>

I'm sorry, this was definitely worded poorly. My intent was to show that
we should not impose any restrictions on content types and we can avoid
a REST-specific media type per the discussion over divergent approaches
to RESTful JSON (see the "Content negotiation" thread). REST is
obviously intended to be usable with various media types, a true REST
architecture should certainly support future JSON subtypes.


>> ...
>> Content-Type: application/project+json; schema="/projectSchema"
>> ...
>> Content-Type: application/schema+json;
>> ...
>> Content-Type: application/myapp+json;
>>
> schema="/projectCollectionSchema"
>
> ...and then you go and invent 3 new content-types. ;)
>

Yes, allowing the use of other subtypes was the point I was (poorly)
trying to make.

> Also, http://tools.ietf.org/html/rfc4288#section-4.2 says:
>
> In accordance with the rules specified in [RFC3023], media subtypes
> that do not represent XML entities MUST NOT be given a name that ends
> with the "+xml" suffix. More generally, "+suffix" constructs should
> be used with care, given the possibility of conflicts with future
> suffix definitions.
>
> So unless you're gunning to be the author of those "future suffix
> definitions" (and it sounds like you might be), be careful with
> "application/*+json" types.
>
>

That's not my goal. I am not sure if JSON Schema should have it's own
media type or not, but my intent was to preserve the future alternate
representation capabilities that are fundamental to REST.

Kris Zyp

unread,
Nov 12, 2008, 3:12:39 PM11/12/08
to Kris Zyp, restfu...@googlegroups.com
Another thought, one could also approach the data-schema relationship
from the other direction. That is, a user could access a schema first,
and then access the data (instances of that schema) as a link from the
schema, where the link relationship is "data" or "collection" . That is
a schema could have:

{
"data":{"href":"/Project/"} # points to the collection of instances of this schema
"metadata": {
...
},
"properties":{
...
},

}
This could easily be added to the schema since it is well-defined
structure. One could even define the forward and reverse relationship in
the schema:

{
"data":{"href":"/Project/", "schema":{"href":"#"}},
...

The advantage of this you wouldn't need to rely MIME-type parameters. However, referencing data instances from the schema and schemas from the data don't necessarily need to be mutually exclusive, and if data relies on a schema, not having a link may be an REST anti-pattern since it would prevent intelligible bookmarkability of data resources.

From a hypertext navigation perspective, starting with a list of schemas as an entry point to JSON-based site makes a lot of sense, it allows users to see the structure and descriptions of the available data and then drill down into the actual data instances from there.
Kris

David Snopek

unread,
Nov 12, 2008, 3:15:45 PM11/12/08
to restfu...@googlegroups.com
2008/11/12 Robert Brewer <fuma...@aminus.org>:
[snip]

> Really? The existence of AtomPub would seem to argue against that. If
> even XML needs separate content-types, why wouldn't JSON?
[snip]

> I'm headed in the other direction. I think we can have a spec like an
> AtomPub without needing a "JSON media types" spec a la RFC 3023, and
> without needing generic JSON schema specs. I'm about 90% done with a new
> media-type along those lines which I'll try to share soon.

We definitely can have new JSON-based media-types and I think we will
see plenty of them soon. I look forward to seeing yours when you
share it.

The problem I think Kris is trying to solve, is a way for user agents
to be able to understand several such media-types without having to
have explicit support. So, maybe one user agent only supports
JSON-based media-type X and Y. Now Z comes along. If it supplied a
schema like Kris is suggesting, then the user agent could atleast do
something basic with it. It might not understand everything, but
atleast its one step above line noise.

I think something like this may become even more necessary soon, as I
see that there are many, many people working on JSON-based
media-types.

Cheers,
David.

Kris Zyp

unread,
Nov 12, 2008, 3:35:11 PM11/12/08
to restfu...@googlegroups.com

> The problem I think Kris is trying to solve, is a way for user agents
> to be able to understand several such media-types without having to
> have explicit support. So, maybe one user agent only supports
> JSON-based media-type X and Y. Now Z comes along. If it supplied a
> schema like Kris is suggesting, then the user agent could atleast do
> something basic with it. It might not understand everything, but
> atleast its one step above line noise.
>
Yes, and this doesn't need to only be applied to different actual
media-types. This can simply be used to describe the different data
structures that people are currently using with plain ol'
application/json that include hypertext and metadata in various shapes
and forms, so that a generic clients/user agents can do hypertext
navigation through JSON data and send updates back to the server. It is
also not intended to be a "universal" media type like those proposed in
the articles referenced by Alexandros, but rather a way provides a
defined mappings between user-defined properties and fixed set of
well-known REST concepts.
Kris

John Kemp

unread,
Nov 12, 2008, 4:40:01 PM11/12/08
to restfu...@googlegroups.com
Kris Zyp wrote:

...

> Yes, and this doesn't need to only be applied to different actual
> media-types. This can simply be used to describe the different data
> structures that people are currently using with plain ol'
> application/json that include hypertext and metadata in various shapes
> and forms, so that a generic clients/user agents can do hypertext
> navigation through JSON data and send updates back to the server. It is
> also not intended to be a "universal" media type like those proposed in
> the articles referenced by Alexandros, but rather a way provides a
> defined mappings between user-defined properties and fixed set of
> well-known REST concepts.

Wouldn't this be possible (as JSON objects are representations of HTTP
resources) using existing resource-oriented linking concepts such as the
HTTP Link header[1]?

For example:

HTTP/1.1 200 OK
Content-Type:...
Link: http://json-schema.org/schema; rel="schema"

{
// metadata:
"type": "application/project+json",
"href": "proposal",
"id": "proposal",
// data:
"data": {
"projectOwner": {
// metadata:
"href": "/Person/kris",
"id":"kris",
// data:
"name":"Kris Zyp"
}
}
}


Regards,

- johnk

[1] http://tools.ietf.org/html/draft-nottingham-http-link-header-02

Subbu Allamaraju

unread,
Nov 12, 2008, 4:48:16 PM11/12/08
to restfu...@googlegroups.com
Possible, but header based linking is less flexible than links
contained within representations. For example, consider a collection
of resources, with each member containing one or more links. It will
be difficult to express those many links as headers.

The link header is useful when the media type chosen for a
representation does not permit linking (e.g. a pdf file or an image),
but the server still wants to provide some links.

Moreover, AFAIK, the header linking I-D is not a done deal.

Subbu
---
http://subbu.org

Kris Zyp

unread,
Nov 12, 2008, 4:50:05 PM11/12/08
to restfu...@googlegroups.com

>
> Wouldn't this be possible (as JSON objects are representations of HTTP
> resources) using existing resource-oriented linking concepts such as
> the HTTP Link header[1]?
>
> For example:
>
> HTTP/1.1 200 OK
> Content-Type:...
> Link: http://json-schema.org/schema; rel="schema"
Cool, that sounds like a good idea. The only drawback I would see is
there may be environments where only a Content-Type is available, like
maybe some attachments, and for header-limited cross-domain XHR under
the cross-site for access control proposal [1]. These may not be serious
drawbacks though, and with CS-XHR you can negotiate access to headers. I
like this header-based approach, +1 from me.
[1] http://www.w3.org/TR/access-control/
Kris

David Snopek

unread,
Nov 12, 2008, 5:15:19 PM11/12/08
to restfu...@googlegroups.com
2008/11/12 John Kemp <jo...@jkemp.net>:
[snip]

> Wouldn't this be possible (as JSON objects are representations of HTTP
> resources) using existing resource-oriented linking concepts such as the
> HTTP Link header[1]?
>
> For example:
>
> HTTP/1.1 200 OK
> Content-Type:...
> Link: http://json-schema.org/schema; rel="schema"
[snip]
>
> [1] http://tools.ietf.org/html/draft-nottingham-http-link-header-02
>

This is a cool I-D! It'll be interesting to see where it goes. If it
becomes an RFC, this is definitely a more appropriate header for this
type of information.

John Kemp

unread,
Nov 12, 2008, 5:28:18 PM11/12/08
to restfu...@googlegroups.com
Subbu Allamaraju wrote:
>
> Possible, but header based linking is less flexible than links contained
> within representations. For example, consider a collection of resources,
> with each member containing one or more links. It will be difficult to
> express those many links as headers.

You are certainly correct, but I wasn't saying that one shouldn't
include such links where appropriate and possible.

>
> The link header is useful when the media type chosen for a
> representation does not permit linking (e.g. a pdf file or an image),
> but the server still wants to provide some links.

Is it not also useful if there is a single representation contained
within the entity-body, and the link is thus related to that specified
resource?

- johnk

Kris Zyp

unread,
Nov 17, 2008, 4:06:49 PM11/17/08
to restfu...@googlegroups.com
I have been thinking about how a hypertext defining schema should be
structured. JSON Schema can be self-descriptive, i.e. a meta-schema
exists that defines the structure of schemas [1]. Likewise, we can have
a meta-schema for "hyper-schemas" that defines the structure schema that
provide hypertext mapping. However, as I was thinking about this, I
believe that we would want minimize the number of properties that are
ad-hoc mapped. I believe that based on existing usage of JSON in RESTful
scenarious, at the very least we do want to allow user-defined hyperlink
properties (properties that map to URLs), and I think there are several
others as well. However, there a number of properties that rarely ever
represented latently in data. Properties like media-type, etags, and so
forth could be a part of actual predefined JSON structure that is itself
described as a hyper-schema.

Here is my proposed definition for hyper-schemas, written as a
meta-circular hyper-schema itself (and consequently it's schema refers
to itself):

Content-Type: application/json; schema="#"
{
"extends": {"href":"http://json-schema.org/schema"},
"description": "A schema for schemas that define how hypertext
information is interpreted from instance objects",
"schema": {"href":"#"},
"hrefProperty":"href",
"properties":{
"hrefProperty":{
"type":"string",
"description":"This defines the name of the property that
indicates a hyperlink. When this property is encountered in the instance
object or any of it's child objects, the value of the property indicates
target URI of the hyperlink and is resolved per RFC 1808. The object may
include other properties providing a partial/summary representation of
the target resource",
"optional":true
}
"locatorProperty":{
"type":"string",
"description":"This defines the name of the property that
indicates the location of the object. When this property is encountered
in the instance object or any of it's child objects, the object
represents a resource, whose URI is indicated by the value of the
property and is resolved per RFC 1808. The object is treated as a full
representation of the target resource",
"optional":true
},
"data":{
"type":"array",
"description":"This should refer to the collection of
instances of this defined schema"
},
"baseUrl":{
"type":"string",
"description":"This defines the base URL to be used for
resolving hyperlinks and locator property values. When this property is
not present, the URL that was used to retrieve the representation of
resource is used",
"optional":true
},
"envelopeFor":{
"type":"string",
"description":"This presence of this property indicates that
the instance object is an envelope for the real representation of the
resource. This defines the name of property that refers to the actual
data of the real representation. All other properties of the instance
object are considered metadata"
"optional": true
},
"queryEncoding":{
"type":"string",
"description":"This property indicates the query encoding
format that should be used for querying the collection of instances. The
query can be postpended to the collection URL to query the collection",
"options":[
{"name":"Form URL Encoded", "value":
"application/x-www-form-urlencoded"},
{"name":"JSONQuery URL Encoded", "value":
"application/x-jsonquery-urlencoded"}
],
"optional": true,
"default": "application/x-www-form-urlencoded"
},
"versionProperty":
"type":"string",
"description":"This property indicates the version of the
instance. Any modification to the instance must cause a change in the
version value",
"optional": true
}
}
}

There are several important aspects of navigation and interaction
guidance that user-agents can ascertain from a schema:
1. Where to access a collection of the instances.
2. How to determine what property defines a hyperlink.
3. The properties to expect and a description of those meanings (useful
when showing a listing of the instances for the human user, and creating
forms).
4. How to query the collection of products based on criteria. This
schema defines the properties of the objects, and the queryFormat
describes the format of the query. This is analogous HTML forms which
provide an interface for the user to generate a query.
5. If the user wants to update data or create new instances/resources,
it knows the format the server expects.

Finally, we could create a "recommended" hyper-schema that follows the
meta-schema specification and provides RESTful metadata information.
User-agents may wish to comprehend parts of this schema as well, and
user could still conveniently extend this schema for defining their own
data structures on top of the metadata spec. Users could then choose to
create their own hyper-schema (not extending anything) for their custom
data structures, or they could choose to use (or extend) the recommended
hyper-schema with it's extra meta-data information. Hopefully, Robert
Brewer's proposal (I don't know for sure what that will look like) could
be described with a hyper-schema and could become the recommended
structure for including metadata. This would create a very nice
consistency between custom and spec recommended data structures. Here is
a rough example of hyper-schema that would describe (some of) John
Cowan's JSON Web Collections proposed structure:

Content-Type: application/json; schema="http://json-schema.org/hyper-schema"
{
"description": "A schema for JSON Web Collections",
"schema": {"href":"http://json-schema.org/hyper-schema"},
"hrefProperty": "href",
"envelopeFor": "members",
"properties":{
"version":{
"description":"identifies this version of the collection"
},
"type":{
"description":"the media type of the members of this collection"
}
},
"items":{
"envelopeFor": "value",
"properties":{
"precis":{
"description":"A summary of the item"
}
}
}
}

But at the same time, existing servers/implementations could easily
create a hyper-schema to describe their data for user agents. For
example, a CouchDB hyper-schema could be created that would enable
user-agents to understand the CouchDB collection format:
{
"description":"A CouchDB set of data",
"hrefProperty":"id",
"versionProperty":"rev",
"envelopeFor":"rows",
"items":{
"locatorProperty":"_id"
}
}

I thought it might be interesting to look at an example from the uniform
interface hypertext navigation perspective and how a user could navigate
through a JSON site in an intelligible way. A fun example, where we
start with no a priori knowledge, no API (except assuming the user agent
understands hyper-schemas media), just a URL (the root URL, naturally):
GET /
Accept: application/json

response:
Link: #schema; rel="schema"
{
"description":"Welcome to Joe's Toy Store",
"products": {
"hrefProperty":"href",
"description":"Our great selection of toys",
"properties":{
"name":{
"type":"string",
"description":"Name of the toy"
},
"price":{
"type":"number"
}
},
"data":{"href":"/Toy/"},
"queryEncoding":"application/jsonquery"
},
"orders":{
"hrefProperty":"href",
"description":"Place your order before Christmas",
"properties":{
"product":{"href":"#products"},
"price":{
"type":"number"
}
},
"data":{"href":"/Order/"}
}
"schema":{
"description":"Site Description",
"properties":{
"products":{"href":"http://json-schema.org/hyper-schema"},
"orders":{"href":"http://json-schema.org/hyper-schema"}
}
}
}

Once again, the user/user agent learns a number of things, specifically
applied to this situation:
1. Where to access a collection of the products offered by the store (at
/Toy/).
2. How to determine what property defines a hyperlink (just "href" in
this case).
3. The properties to expect and a description of those meanings (useful
when showing a listing of the products for the human user, and creating
forms):
4. How to query the collection of products based on criteria. This
schema defines the properties of the objects, and the queryFormat
describes the format of the query. This is analogous HTML forms which
provide an interface for the user to generate a query.
5. If the user wants to update data or create new instances/resources,
it knows the format the server expects. If the user had permission to
update product information they could do so. We also now know how to
POST a new order.

And back to following the user navigation:
GET /Toy/

response:
[{"href":"slinky","name":"Slinky"},
{"href":"barbie","name":"Barbie"}]

User takes a look at the barbie:
GET /Toy/barbie

response
{"href":"barbie","name":"Barbie","price":9.95,"color":"pink"}

This is too expensive, so the user wants to narrow the search based on
price. The user agent has enough information from the schema to provide
a form/UI for creating a query. The user chooses that products under $5:
GET /Toy/[?price<5]

response:
[{"href":"slinky","name":"Slinky"}]

The user decides that he will give slinkies to his children for
Christmas. The user agent has sufficient information to provide a form
for the POSTing an the order. The user chooses a quantity of 3 and the
slinky product:
POST /Order/
{
"product":{"href":"/Toy/slinky"}
"quantity":3
}

response status code of 201. Order completed. Next, the site owners
decide that maybe they should require a credit card number and address
as well.

The point of the example is that the user agent can navigate and
interact with this site using JSON and following the schema and
hypertext information found it.
Kris


[1] http://json-schema.org/schema

Robert Brewer

unread,
Nov 17, 2008, 5:27:02 PM11/17/08
to restfu...@googlegroups.com
Kris Zyp wrote:
> Finally, we could create a "recommended" hyper-schema that follows the
> meta-schema specification and provides RESTful metadata information.
> User-agents may wish to comprehend parts of this schema as well, and
> user could still conveniently extend this schema for defining their
own
> data structures on top of the metadata spec. Users could then choose
to
> create their own hyper-schema (not extending anything) for their
custom
> data structures, or they could choose to use (or extend) the
> recommended hyper-schema with it's extra meta-data information.
> Hopefully, Robert Brewer's proposal (I don't know for sure what that
> will look like) could be described with a hyper-schema

It probably could, but I'd very much hope it would not. I like JSON for
its "flat is better than nested" approach applied to Abstraction itself,
and would rather not see any more "meta"-anythings than absolutely
necessary. Experience tells me every additional meta-layer reduces the
set of developers who understand a system by at least 90%. Add them with
care.

> There are several important aspects of navigation and interaction
> guidance that user-agents can ascertain from a schema:
> 1. Where to access a collection of the instances.
> 2. How to determine what property defines a hyperlink.
> 3. The properties to expect and a description of those meanings
> (useful when showing a listing of the instances for the human user,
> and creating forms).
> 4. How to query the collection of products based on criteria. This
> schema defines the properties of the objects, and the queryFormat
> describes the format of the query. This is analogous HTML forms which
> provide an interface for the user to generate a query.
> 5. If the user wants to update data or create new instances/resources,
> it knows the format the server expects.

A lot of the benefits of your meta-schema proposal could be better
addressed IMO with a very small set of new media-types which actively
reduce the choice of structures and behaviors one uses with JSON, rather
than trying to expand them or bless them all with a JSON-linguistics.
Not to beat a dead horse, but Atom+AtomPub achieves the above by 1)
constraining method semantics, e.g. what POST on a Collection does, 2)
atom:link, 3) limited, specific objects, such as app:collection or
atom:feed, 4) plain ol' hyperlinks, and 5) nothing special. I'll be the
first to agree that it could include better support for (4) with
something like URI Templates and (5) with resource templates.

In this sense, the Shoji protocol (for which I have a complete spec, but
don't have permission to share in full yet), is somewhat to JSON what
Atom/APP is to XML. However, it will be quite different in that Shoji is
not a syndication format/protocol, but rather what I'm calling a
"catalog" format/protocol; that is, something which is appropriate for
OLTP data. Atom is designed for time-ordered documents and is heavy on
synchronization use cases. Shoji is designed for nested sets of data and
heavy on fractional-update use cases. We might need both, but I have no
problem right now using Atom for syndication, and pushing Shoji for what
I'm calling "catalogs". I think it does almost all of what Joe Gregorio
asked for [1].

Shoji defines three document types (I haven't decided yet whether to
make 3 actual media-types or 1 multivariate one):

* Values: any JSON, including tabular data "views" as arrays of arrays.
* Entities: "relations", or sets of Values. Implemented as JSON
objects.
* Catalogs: sets of relations. Implemented as JSON objects, which
behave very much like AtomPub Collections; for example, POST'ing an
Entity message to a Catalog adds the entity to the collection.

Entities and Catalogs have plenty of hyperlinks between them, including
$self, plus IRI Patterns based on Fielding's templates [2]. The entire
ABNF is ~26 productions right now. So far, I've been able to very easily
design the messages for our entire /users tree as Shoji documents. That
includes auth, by the way, and all of it uses stored procedures, not an
ORM.

I'm *not* saying there's no place for meta-JSON proposals. I just wanted
to say "my proposal probably doesn't need it", and abuse the opportunity
to give a better glimpse of what Shoji will be. I'd personally rather
build 3 or 4 processors to handle 3 or 4 new targeted, JSON-related
media types than one metaprocessor to rule them all. IMO the
metaprocessing approach hasn't worked out all that well in the XML
world, or any other I'm aware of.


Robert Brewer
fuma...@aminus.org

[1] http://bitworking.org/news/358/restful-json
[2] http://lists.w3.org/Archives/Public/uri/2008Sep/0007.html

Kris Zyp

unread,
Nov 17, 2008, 6:33:07 PM11/17/08
to restfu...@googlegroups.com

Robert Brewer wrote:


> Kris Zyp wrote:
>
>> data structures, or they could choose to use (or extend) the
>> recommended hyper-schema with it's extra meta-data information.
>> Hopefully, Robert Brewer's proposal (I don't know for sure what that
>> will look like) could be described with a hyper-schema
>>
>
> It probably could, but I'd very much hope it would not. I like JSON for
> its "flat is better than nested" approach applied to Abstraction itself,
> and would rather not see any more "meta"-anythings than absolutely
> necessary. Experience tells me every additional meta-layer reduces the
> set of developers who understand a system by at least 90%. Add them with
> care.
>

Once again, I think my wording was unclear here. To be clear, I think it
would be beneficial if Shoji's structure and hypertext mechanisms could
be described with a JSON hyper-schema, but I certainly don't think that
means that all the meaning of Shoji should or could be mechanically
communicated by a JSON hyper-schema. I presume that Shoji will
communicate many concepts that carry much more meaning to a user agent
than simply that it is a string or hyperlink (what would be understood
from a hyper-schema). While I suppose it would be conceivable to try to
make JSON hyper-schema's capable of mapping every concept from Shoji to
user-defined properties, I am thinking that is going too far.
Meta-definitions are indeed more complicated for users, I certainly
agree with that. However, the important thing is to understand what a
user may need to be able to configure (and the state of implementations
seems to prove that a hyperlink property is one), and what is simpler to
pre-define in a structured specification like Shoji. I think it is naive
to assume all applications and their structures deserve the same treatment.

I really believe what we are working towards is complementary. JSON
hyper-schemas would be valuable for describing data structures where
URLs are mapped to given properties, and Shoji provides a more
structured format for communicating deeper meaning around the resources
being represented. I think there is a strong synergy that can be
leveraged here. If Shoji had a hyper-schema definition, a user agent
that understood hyper-schemas, could partially understand the meaning of
the Shoji data, being able to navigate through links and interact with
the server. A full Shoji-aware client would then have enhanced
capabilities. Furthermore, the schemas would provide a natural way of
users communicating the structures of the "Values" within the Shoji
structure.

> Shoji defines three document types (I haven't decided yet whether to
> make 3 actual media-types or 1 multivariate one):
>
> * Values: any JSON, including tabular data "views" as arrays of arrays.
> * Entities: "relations", or sets of Values. Implemented as JSON
> objects.
> * Catalogs: sets of relations. Implemented as JSON objects, which
> behave very much like AtomPub Collections; for example, POST'ing an
> Entity message to a Catalog adds the entity to the collection.
>
> Entities and Catalogs have plenty of hyperlinks between them, including
> $self, plus IRI Patterns based on Fielding's templates [2]. The entire
> ABNF is ~26 productions right now. So far, I've been able to very easily
> design the messages for our entire /users tree as Shoji documents. That
> includes auth, by the way, and all of it uses stored procedures, not an
> ORM.
>

This sounds great, and I am looking forward to seeing Shoji!


>
> I'm *not* saying there's no place for meta-JSON proposals. I just wanted
> to say "my proposal probably doesn't need it", and abuse the opportunity
> to give a better glimpse of what Shoji will be.

Agreed, Shoji doesn't need a schema. But like I was saying, I don't
think they need to be at odds with each other, hyper-schemas and Shoji
could be complement each other very well. I am certainly hoping our work
is convergent and compatible.

Kris

David Snopek

unread,
Nov 18, 2008, 2:54:47 PM11/18/08
to restfu...@googlegroups.com
2008/11/17 Kris Zyp <kri...@gmail.com>:

>
>> I'm *not* saying there's no place for meta-JSON proposals. I just wanted
>> to say "my proposal probably doesn't need it", and abuse the opportunity
>> to give a better glimpse of what Shoji will be.
>
> Agreed, Shoji doesn't need a schema. But like I was saying, I don't
> think they need to be at odds with each other, hyper-schemas and Shoji
> could be complement each other very well. I am certainly hoping our work
> is convergent and compatible.

I concur, these types of efforts are/should be complimentary. Like
Robert and many others, I am also developing a RESTful protocol with
its own JSON-based media-type.

When all these new protocols and media-types hit the open web, it
would be nice if there were some way in which they could all be
compatible, even at the most basic level (envelopes, hyperlinks, ids,
etc) *without* having to comprise the design of the protocols and
media-types themselves. Like Kris says, Shoji will still be Shoji.
It doesn't need the schema. But with one, user agents that don't
speak Shoji (but do understand the hyper-schema) could communicate
with a Shoji resource on a basic level.

This is a much more practical plan, than either: (1) expecting
everyone to drop their protocol/media-types in favor of some single
true protocol/media-type, or (2) expecting all protocols/media-types
to use the same conventions for envelope (or lack there of),
hyperlinks, ids and so on.

Having some hyper-schema standard would allow everyone to keep
developing their own independent protocols/media-types while giving
user agents a break.

Happy Hacking!

Robert Brewer

unread,
Nov 18, 2008, 3:24:46 PM11/18/08
to restfu...@googlegroups.com
David Snopek wrote:
> 2008/11/17 Kris Zyp <kri...@gmail.com>:
> >
> >> I'm *not* saying there's no place for meta-JSON proposals. I just
> wanted
> >> to say "my proposal probably doesn't need it", and abuse the
> opportunity
> >> to give a better glimpse of what Shoji will be.
> >
> > Agreed, Shoji doesn't need a schema. But like I was saying, I don't
> > think they need to be at odds with each other, hyper-schemas and
> Shoji
> > could be complement each other very well. I am certainly hoping our
> work
> > is convergent and compatible.
>
> I concur, these types of efforts are/should be complimentary. Like
> Robert and many others, I am also developing a RESTful protocol with
> its own JSON-based media-type.
>
> When all these new protocols and media-types hit the open web, it
> would be nice if there were some way in which they could all be
> compatible, even at the most basic level (envelopes, hyperlinks, ids,
> etc) *without* having to comprise the design of the protocols and
> media-types themselves. Like Kris says, Shoji will still be Shoji.
> It doesn't need the schema. But with one, user agents that don't
> speak Shoji (but do understand the hyper-schema) could communicate
> with a Shoji resource on a basic level.

Maybe I've been writing servers too long and not enough clients. What could a user agent possibly do with e.g. a Shoji resource if it doesn't speak Shoji? I can't think of a single useful thing that's not already covered by "it's JSON, so a JSON parser won't choke on it". Okay, maybe it can detect the URI's in a message, but...so what? It's only the specific media type that gives those URI's any meaning.

I don't understand what this "basic level" is that keeps being thrown around without seeing concrete examples (preferably in psuedocode).


Robert Brewer
fuma...@aminus.org

Kris Zyp

unread,
Nov 18, 2008, 4:22:58 PM11/18/08
to restfu...@googlegroups.com

Robert Brewer wrote:
> David Snopek wrote:
>
>> 2008/11/17 Kris Zyp <kri...@gmail.com>:
>>

>>> is convergent and compatible.
>>>
>> I concur, these types of efforts are/should be complimentary. Like
>> Robert and many others, I am also developing a RESTful protocol with
>> its own JSON-based media-type.
>>
>> When all these new protocols and media-types hit the open web, it
>> would be nice if there were some way in which they could all be
>> compatible, even at the most basic level (envelopes, hyperlinks, ids,
>> etc) *without* having to comprise the design of the protocols and
>> media-types themselves. Like Kris says, Shoji will still be Shoji.
>> It doesn't need the schema. But with one, user agents that don't
>> speak Shoji (but do understand the hyper-schema) could communicate
>> with a Shoji resource on a basic level.
>>
>
> Maybe I've been writing servers too long and not enough clients. What could a user agent possibly do with e.g. a Shoji resource if it doesn't speak Shoji? I can't think of a single useful thing that's not already covered by "it's JSON, so a JSON parser won't choke on it". Okay, maybe it can detect the URI's in a message, but...so what? It's only the specific media type that gives those URI's any meaning.
>
> I don't understand what this "basic level" is that keeps being thrown around without seeing concrete examples (preferably in psuedocode).
>
>

I think that's certainly the important question to answer. I attempted
to describe through example how a user-agent could have meaningful
interaction with the data based on the information provided by a
hyper-schema in the example of "Joe's Toy Store" at the bottom of my
message:
http://groups.google.com/group/restful-json/msg/04fb42cbf4972da4. Here
the schema provides enough information that the user-agent can follow
links (including where to find the collections and where to find more
detailed information on a product), perform queries, modify data, and
find text-based descriptions of the properties, and conform to the
structure of the data for the creation of new resources. How that data
is mapped to a UI is obviously somewhat open for interpretation; JSON is
not a visual markup format. However, as an exercise in REST
architecture, a generic "JSON browser" user-agent would have enough
information to provide a real interface for navigating through and
purchasing products from Joy's store, (albeit probably ugly since all it
would know would a set of JSON values). The user-agent would be able to
provide a user the option of following a link to products, and then
following a link to fuller description of a product, and providing a
form for querying the collection. They would also have the appropriate
information for creating a new "order" resource as well through a form.
This is all analogous to HTML's capabilities sans the visual markup
information.

Of course no one is really going to solely provide their data for some
theoretical "JSON browser". A general JSON browser may be useful for
development and prove REST conformance, but would surely be ugly. All
JSON data is ultimately going to be fronted by some UI. Of course the
REST architecture is intended to provide programmatic benefits as well,
and the abstractions that it provides are valuable as well. I will speak
from the perspective of Dojo's implementation of a REST client (called
JsonRestStore). JsonRestStore provides a framework layer that allows the
application layer to interact with the server via a data abstraction,
called Dojo Data. The Dojo Data interface was developed long before the
JsonRestStore and provides a standard way of querying, get properties
from items, creating new items, and modifying items, and so forth. Many
of Dojo's widgets are built to interface with this API. The
JsonRestStore fits perfectly with this API, application code can work at
the data layer, issuing queries, and then looking at item's properties
and modifying them, and JsonRestStore translates these data level
requests to HTTP requests. For example if we had an item (received from
the server with a JSON representation), we can do things like:
store.getValue(item,"foo") -> get's the foo property value
And if the property value is actually a hyperlink/reference, the
JsonRestStore will indicate that the target item is not loaded, and when
loadItem is called it will handle loading the target of it the
reference, and providing the full representation to the application
code. Likewise, one could modify properties:
store.setValue(item,"foo","bar")
And the JsonRestStore will handle translating this modification into a
PUT request to the server (based on the JSON object's identification of
it's URI based on a locator property). The application code doesn't need
to interact with the server at the HTTP level, but rather has the
advantage of working at the data level (regardless of the actual
structure of that data).

Currently the JsonRestStore handles it's interpretation of
references/hyperlinks, object URL mapping, and defining queries either
through configuration settings (or hard-coded properties like "$ref") or
subclasses that alter behavior (for example there is a CouchDBRestStore
that extends the JsonRestStore), not through schemas. Of course this is
certainly a REST anti-pattern, the REST architecture would suggest that
the user agent can determine all the relevant information needed for
navigation based on the server response's. If the JsonRestStore could do
all these configuration's based on schemas provided by the server, there
would only need to be a single class and it could "configure" itself.

Does that help clear up what is the "basic level"?
Kris


Robert Brewer

unread,
Nov 19, 2008, 4:14:28 AM11/19/08
to restfu...@googlegroups.com
You're right that these are analogues of HTML capabilities, and any new protocol should address them. But I wonder why you would then advocate deviating from the design of HTML, which places the 5 things you mentioned clearly in the prose of a concrete media-type (HTML), not in the grammar of a meta-schema (either SGML or the design of DTD's themselves, depending on how you're thinking of "meta"):

> 1. Where to access a collection of the instances.
> 2. How to determine what property defines a hyperlink.

SGML defines the "URI" entity, yes, although it defers the semantics of the URI datatype entirely to RFC 2396. But it is the concrete media-type, HTML, which defines the LINK and A entities, and the various LinkTypes. The design of DTD's themselves does neither, nor does, say, XMLSchema: it only defines lexical properties of the anyURI datatype such as 'cardinality' and 'length'; semantic properties are left to media types (possibly expressed using XMLSchema).

> 3. The properties to expect and a description of those meanings
> (useful when showing a listing of the instances for the human user,
> and creating forms).
> 4. How to query the collection of products based on criteria. This
> schema defines the properties of the objects, and the queryFormat
> describes the format of the query. This is analogous HTML forms
> which provide an interface for the user to generate a query.

These are implemented in HTML's FORM entity and associated INPUT controls, plus LABEL etc. Again, a concrete media-type.

> 5. If the user wants to update data or create new instances/resources,
> it knows the format the server expects.

In HTML-land, it knows the format only insofar as the FORM elements tell it via "enctype" attributes (= media types) and a limited inkling of how to encode form values into form submissions for only 2 media types: application/x-www-form-urlencoded and multipart/form-data.

In a JSON analogue, I would expect to see the concrete media-type, not a schema spec, define all of these 5 things. Why do you feel putting them in a schema spec is better? Did SGML/HTML/XML fail in this respect? My fear is that your proposed meta-schema will end up being just another media type in all but name; sort of an end-run around useful, specific, concrete media-types in an attempt to produce a generic, abstract media-type (that nobody will find useful except Dojo).

> ...the JsonRestStore will handle translating this modification into a
> PUT request to the server (based on the JSON object's identification of
> it's URI based on a locator property). The application code doesn't
> need to interact with the server at the HTTP level, but rather has
> the advantage of working at the data level (regardless of the actual
> structure of that data).

You might guess that I, as the lead dev of CherryPy, might balk at attempts to abstract away HTTP ;)

> Currently the JsonRestStore handles it's interpretation of
> references/hyperlinks, object URL mapping, and defining queries either
> through configuration settings (or hard-coded properties like "$ref")
> or subclasses that alter behavior (for example there is a CouchDBRestStore
> that extends the JsonRestStore), not through schemas. Of course this is
> certainly a REST anti-pattern, the REST architecture would suggest that
> the user agent can determine all the relevant information needed for
> navigation based on the server response's.

Indeed, and that information is baked into concrete media types. I find it odd that Persevere speaks so strongly about JSON Schema yet makes almost no mention of media types; IMO the entire section on referencing in the JSON Schema spec should be eliminated and left to media types built using the schema. From my point of view, your proposal mixes lexical, syntactic, and semantic analysis far too strongly. IMO the proper place for defining URI syntax and semantics is in a document media type, not a schema (although a schema certainly may define URI semantics for its own grammar to aid the act of producing a media type (as opposed to the grammar of the produced media type), e.g. RELAX NG's externalRef).

> If the JsonRestStore could do all these configuration's based on
> schemas provided by the server, there would only need to be a
> single class and it could "configure" itself.

Yes, that would be a nice code reduction. But it wouldn't loosen the coupling between Dojo and media types.


Robert Brewer
fuma...@aminus.org

Kris Zyp

unread,
Nov 19, 2008, 10:01:56 AM11/19/08
to restfu...@googlegroups.com

It comes down to the basic fundamental difference between the structure
of HTML/XML and JSON and how people use these formats to represent their
data/resources. XML has plenty of "room" for metadata and extra
information about hyperlinks without affecting the flow of the document
that is being marked up. JSON on the other hand, often has unambiguous
precise representation for a given resource and could be semantically
altered if certain structural requirements are placed on it. JSON data
frequently contains existing relationships to data that has been mapped
to URLs, and this should be leveraged, both for preserving accurate
representations as well as size efficiency.

When it is possible to "force" users to follow a certain structure by a
concrete media-type, I certainly agree that is most the convenient and
straightforward approach. But I really don't believe that will fit
everyone who is using JSON in a RESTful architecture.

>> 1. Where to access a collection of the instances.
>> 2. How to determine what property defines a hyperlink.
>>
>
> SGML defines the "URI" entity, yes, although it defers the semantics of the URI datatype entirely to RFC 2396.

And we should certainly defer to RFC 2396 for the semantics of URI,
noting that RFC 2396 explicitly states that media types are responsible
for the interpretation of fragment identifiers. The interpretation of
fragment identifiers in JSON (using JSONPath semantics) has already been
discussed, and I think that is still an important aspect of a RESTful
JSON media type (whether it be via hyper-schemas or something concrete
like Shoji). Anyway, I don't think that is really what you wanted to
argue about though, sorry for the rabbit trail...

>
>> 3. The properties to expect and a description of those meanings
>> (useful when showing a listing of the instances for the human user,
>> and creating forms).
>> 4. How to query the collection of products based on criteria. This
>> schema defines the properties of the objects, and the queryFormat
>> describes the format of the query. This is analogous HTML forms
>> which provide an interface for the user to generate a query.
>>
>
> These are implemented in HTML's FORM entity and associated INPUT controls, plus LABEL etc. Again, a concrete media-type.
>
>

The hyper-schema itself certainly should and would be a concrete
media-type itself, and this where the set of property definitions and
query encoding types are defined. The hyper-schema is "meta" in regards
to the data it describes, but the queries could certainly be formulated
based on the hyper-schema directly.

>> 5. If the user wants to update data or create new instances/resources,
>> it knows the format the server expects.
>>
>
> In HTML-land, it knows the format only insofar as the FORM elements tell it via "enctype" attributes (= media types) and a limited inkling of how to encode form values into form submissions for only 2 media types: application/x-www-form-urlencoded and multipart/form-data.
>

s/format/structure. If you are creating a resource, it is certainly
valuable to understand the structure of the JSON that is necessary (what
properties are expected and what type they are). This is analogous to
the actual INPUTs of a FORM, once again defined in concrete form in the
hyper-schema.


>
>> ...the JsonRestStore will handle translating this modification into a
>> PUT request to the server (based on the JSON object's identification of
>> it's URI based on a locator property). The application code doesn't
>> need to interact with the server at the HTTP level, but rather has
>> the advantage of working at the data level (regardless of the actual
>> structure of that data).
>>
>
> You might guess that I, as the lead dev of CherryPy, might balk at attempts to abstract away HTTP ;)
>
>

There are countless abstractions within any application, is HTTP the one
mechanism that should never be abstracted? I certainly am not suggested
that we should try to keep developers from ever interacting with HTTP, I
wish more developers understood HTTP, but I more just wanted to describe
our abstraction and how our API works with our existing components as an
example.


>> Currently the JsonRestStore handles it's interpretation of
>> references/hyperlinks, object URL mapping, and defining queries either
>> through configuration settings (or hard-coded properties like "$ref")
>> or subclasses that alter behavior (for example there is a CouchDBRestStore
>> that extends the JsonRestStore), not through schemas. Of course this is
>> certainly a REST anti-pattern, the REST architecture would suggest that
>> the user agent can determine all the relevant information needed for
>> navigation based on the server response's.
>>
>
> Indeed, and that information is baked into concrete media types. I find it odd that Persevere speaks so strongly about JSON Schema yet makes almost no mention of media types; IMO the entire section on referencing in the JSON Schema spec should be eliminated and left to media types built using the schema. From my point of view, your proposal mixes lexical, syntactic, and semantic analysis far too strongly. IMO the proper place for defining URI syntax and semantics is in a document media type, not a schema (although a schema certainly may define URI semantics for its own grammar to aid the act of producing a media type (as opposed to the grammar of the produced media type), e.g. RELAX NG's externalRef).
>

I certainly agree that it is important to have a separation of concerns.
I attempted to create a separation between JSON referencing (essentially
defining hyperlinks using "$ref" and "id" properties) and JSON Schema,
but that is perhaps unclear. Hopefully hyper-schemas as a separate spec
that extended the schema spec would make that more well-defined. I
didn't specify a media type name for JSON Schema in the specification.
Do you suggest that I do so? It seems like application/schema+json would
be the most appropriate (prefixed with x- for the time being), I can
certainly add that to the spec. Would that help resolve those issues?

>
>> If the JsonRestStore could do all these configuration's based on
>> schemas provided by the server, there would only need to be a
>> single class and it could "configure" itself.
>>
>
> Yes, that would be a nice code reduction. But it wouldn't loosen the coupling between Dojo and media types.
>

Seems like handling multiple JSON data structures without configuration
could be described as "loosening the coupling", but even if not,
regardless of the wording, that is the goal.

Kris

Robert Brewer

unread,
Nov 19, 2008, 9:06:10 PM11/19/08
to restfu...@googlegroups.com
Kris Zyp wrote:
> Robert Brewer wrote:
> > > This is all analogous to HTML's capabilities sans the visual
> > > markup information.
> >
> > You're right that these are analogues of HTML capabilities, and any
> > new protocol should address them. But I wonder why you would then
> > advocate deviating from the design of HTML, which places the 5 things
> > you mentioned clearly in the prose of a concrete media-type (HTML), not
> > in the grammar of a meta-schema (either SGML or the design of DTD's
> > themselves, depending on how you're thinking of "meta"):
> >
> It comes down to the basic fundamental difference between the structure
> of HTML/XML and JSON and how people use these formats to represent
> their data/resources. XML has plenty of "room" for metadata and extra
> information about hyperlinks without affecting the flow of the document
> that is being marked up. JSON on the other hand, often has unambiguous
> precise representation for a given resource and could be semantically
> altered if certain structural requirements are placed on it.

Getting off track a little bit here (but it's a tangent worth discussing)...

It sounds like you're describing JSON messages as purely an object serialization format, so that a given struct on, say, a server, can be encoded unchanged for shipping to a remote client which then decodes it. Which is pretty close to the canonical explanation for JSON.

However, by itself, it falls short of being RESTful in two important aspects: "self-descriptive messages", and "hypermedia as the engine of application state". There are several ways to provide these:

1. Shoji addresses the former by requiring a "$self" member (an IRI), and the latter by recommending $catalogs, $entities, $fragments, and $views members (lists of IRI's). In Shoji, these metadata elements are mixed into the representation itself, and marked off with the leading "$" character.
2. Other API's might define a format that separates metadata from data yet in the same message (like HTML's HEAD and BODY elements do).
3. Still others might choose to invent new HTTP headers in which to place metadata. Cf the push for a new generic "Link" header in HTTP [1].
4. Yet another might decide that "metadata" is really just a different resource's "data" that should reside at a separate URI. For example, when POSTing a new Entity in Shoji, it might help to have either a schema or a prototype object at a separate URI instead of embedded in full in the catalog entity.

I think you're going to be hard-pressed to invent a "hyper-schema" that covers them all (but you're welcome to try).

> When it is possible to "force" users to follow a certain structure by a
> concrete media-type, I certainly agree that is most the convenient and
> straightforward approach. But I really don't believe that will fit
> everyone who is using JSON in a RESTful architecture.

So it sounds to me like you're an advocate of (3) and maybe (4), then. Personally, I'd rather stick with (1) or (2), because they fulfill the goal of "self-descriptive" better IMO. Will it work for everyone? Probably not. Maybe eighty percent. Do media-types that pick 1 or 2 tend to gain more adherents than 3 or 4? Seems like it to me, looking at other media types. But perhaps that's just an accident of history.

> The hyper-schema itself certainly should and would be a concrete
> media-type itself, and this where the set of property definitions and
> query encoding types are defined. The hyper-schema is "meta" in regards
> to the data it describes, but the queries could certainly be formulated
> based on the hyper-schema directly.
> ...
>
> > IMO the proper place for defining URI syntax and semantics is in
> > a document media type, not a schema...
>
> I certainly agree that it is important to have a separation of
> concerns...I didn't specify a media type name for JSON Schema in
> the specification. Do you suggest that I do so? It seems like
> application/schema+json would be the most appropriate (prefixed
> with x- for the time being), I can certainly add that to the spec.

It would certainly be a welcome addition to label JSON Schema documents with a media-type.

> Would that help resolve those issues?

Perhaps. That would help my server serve application/x-schema+json documents, but IMO they're not needed in order to serve the "real" data, which is application/shoji.

Having given it a lot of thought, I don't think your proposal is really at odds with Shoji; nothing would stop an implementer of Shoji from adding their own "$schema" member with either an IRI to a separate resource (in application/schema+json format) or an embedded schema object. But I'm going to stop short of requiring it in my spec; I don't see the benefit of tying them together. Nor am I inclined to recommend "hrefProperty" and the like (as you did in the Toy Store example) in Shoji messages when my media type already defines what object members are expected to be IRI's. I'd rather see schemas developed in parallel for a while by those who need them, and a winning format emerge based on its adoption by applications, not its adoption by media types. Sorry if I derailed your schema train by assuming tight coupling that doesn't need to exist.


Robert Brewer
fuma...@aminus.org


[1] http://www.mnot.net/drafts/draft-nottingham-http-link-header-00.txt

Kris Zyp

unread,
Nov 21, 2008, 1:26:35 PM11/21/08
to restfu...@googlegroups.com
I essentially agree with most of what you are saying, but had a question
inline...

Robert Brewer wrote:
> Kris Zyp wrote:
>
>> Robert Brewer wrote:
>>
>>>> This is all analogous to HTML's capabilities sans the visual
>>>> markup information.
>>>>
>>> You're right that these are analogues of HTML capabilities, and any
>>> new protocol should address them. But I wonder why you would then
>>> advocate deviating from the design of HTML, which places the 5 things
>>> you mentioned clearly in the prose of a concrete media-type (HTML), not
>>> in the grammar of a meta-schema (either SGML or the design of DTD's
>>> themselves, depending on how you're thinking of "meta"):
>>>
>>>
>> It comes down to the basic fundamental difference between the structure
>> of HTML/XML and JSON and how people use these formats to represent
>> their data/resources. XML has plenty of "room" for metadata and extra
>> information about hyperlinks without affecting the flow of the document
>> that is being marked up. JSON on the other hand, often has unambiguous
>> precise representation for a given resource and could be semantically
>> altered if certain structural requirements are placed on it.
>>
>
> Getting off track a little bit here (but it's a tangent worth discussing)...
>
> It sounds like you're describing JSON messages as purely an object serialization format, so that a given struct on, say, a server, can be encoded unchanged for shipping to a remote client which then decodes it. Which is pretty close to the canonical explanation for JSON.
>

I certainly recognize that this is not the only use of JSON, but it is
an important usage of JSON, it is the primary way that I utilize JSON
and so I would certainly like to be able to preserve the accuracy of the
serialization for these situations, while still leveraging hypertext
(which in the object realm where I am working essentially equates to
lazy loading of properties).

> However, by itself, it falls short of being RESTful in two important aspects: "self-descriptive messages", and "hypermedia as the engine of application state". There are several ways to provide these:
>
> 1. Shoji addresses the former by requiring a "$self" member (an IRI), and the latter by recommending $catalogs, $entities, $fragments, and $views members (lists of IRI's). In Shoji, these metadata elements are mixed into the representation itself, and marked off with the leading "$" character.
> 2. Other API's might define a format that separates metadata from data yet in the same message (like HTML's HEAD and BODY elements do).
> 3. Still others might choose to invent new HTTP headers in which to place metadata. Cf the push for a new generic "Link" header in HTTP [1].
> 4. Yet another might decide that "metadata" is really just a different resource's "data" that should reside at a separate URI. For example, when POSTing a new Entity in Shoji, it might help to have either a schema or a prototype object at a separate URI instead of embedded in full in the catalog entity.
>
> I think you're going to be hard-pressed to invent a "hyper-schema" that covers them all (but you're welcome to try).
>

Yes, I agree. The intent of hyper-schema is certainly more focused on
the HATEOAS aspect. I am curious what is necessary for a
self-descriptive messages? The Fielding thesis describes it as stateless
messages using standard methods, media types, caching metadata so
intermediaries can operate on the messages. By refraining from using
contextual state (like sessions) and leveraging a RESTful transport
(like HTTP), how do plain JSON + HTTP messages fail to be
self-descriptive messages? Or does self-descriptive infer higher-level
meaning (like structural information provided by schemas)? Or is there
specific metadata that is necessary to be self-descriptive?

Anyway, I agree with your points, and hyper-schema and Shoji really
aren't at odds with each other, we just basically have a different
perspective on the expected needs of users. I agree that something Shoji
is probably a better fit for certain needs, but like you were saying,
adoption may advise us on the most important needs.

Kris

Kris Zyp

unread,
Jan 2, 2009, 11:44:15 AM1/2/09
to restfu...@googlegroups.com, json-...@googlegroups.com
I posted a short draft of the JSON hyper-schema proposal here:
http://www.json.com/specifications/hyper-schema/
This proposal is intended to provide a media type definition for a
schema that can define the meaning of properties for hypertext-enabling
JSON. This draft was based on the feedback from the restful-json group
from the discussion around schema-driven description of JSON media for
the purposes of RESTful interaction. Let me know if you have any
thoughts on it. I will probably try to do some implementation work on it
if looks reasonable.
Thanks,

Robert Brewer

unread,
Jan 2, 2009, 1:36:26 PM1/2/09
to restfu...@googlegroups.com, json-...@googlegroups.com
Kris Zyp wrote:
> I posted a short draft of the JSON hyper-schema proposal here:
> http://www.json.com/specifications/hyper-schema/

From that page:
> "baseUrl - This defines the base URL to be used"

Is that right? Or should it say "this defines the name of the property
that indicates the base URL to be used..." Ditto for some others.

> "queryEncoding - This property indicates the query encoding format


that should be used for querying the collection of instances. The query

can be post-pended to the collection URL to query the collection."

An example would be nice. "the...format" is pretty vague. Is that
restricted to Query By Example? Or are queryEncoding formats defined in
the JSON Schema spec?

> "If the user wants to update data or create new instances/resources,

the client knows the format the server expects."

I don't see how that follows from the attributes given.

> "When a fragment identifier is a consists solely a simple token, where
a token is:"

That's almost English ;)

> "token = 1*<any CHAR except CTLs or separators>"

Also the single-quoted-string def. I know you wrapped those in pre tags,
but the parts between angle brackets aren't being rendered. And the
smart quotes make it really hard to read :)

> "If the reference starts with # instead of object location, the
current JSON resource should be considered the starting point for the
property reference."

I think you can just refer to RFC 2396 Sec 5 for all that.

> #foo["another prop"]

I think you mean #foo["another%20prop"]. But I don't see why that would
ever be preferred over #foo.another%20prop...? If the bracket syntax
were reserved for integer indexing, that would make things much simpler.

In general, I like using fragments for this sort of referencing. But I
worry that we are going to end up implementing Javascript inside the
schema spec. Or XPath. Either would be extreme IMO.


Robert Brewer
fuma...@aminus.org

Kris Zyp

unread,
Jan 2, 2009, 2:17:28 PM1/2/09
to restfu...@googlegroups.com
Thanks for the corrections and suggestions. A few comments inline.

Robert Brewer wrote:
> Kris Zyp wrote:
>
>> I posted a short draft of the JSON hyper-schema proposal here:
>> http://www.json.com/specifications/hyper-schema/
>>
>
> From that page:
>
>> "baseUrl - This defines the base URL to be used"
>>
>
> Is that right? Or should it say "this defines the name of the property
> that indicates the base URL to be used..." Ditto for some others.
>

Or maybe baseUrl should just be dropped. Maybe this would be best left
for a media type / schema (not a meta-schema like hyper-schemas, but a
real hard schema). The intention here is to be able to describe various
JSON data structures that may contain existing property values with
meaningful mappings to URLs and other RESTful information. As you have
pointed before, meta-configuration is added complexity, and so we should
certainly avoid meta-configuration of properties if it unlikely that we
need configurability. It seems that there is a lot of evidence of
different real-world data structures contain data that can map to links,
but perhaps it is unlikely that people have existing data properties
that hold base URL information. I do want to leave metadata definitions
that don't require configurability in the domain of a media type like
Shoji and keep hyper-schema minimal.

The actual intent here was that you if someone had data that looked like:
GET /Recipe/apple-pie

{
"name":"Apple Pie",
"ingredients":[{"href":"apples"},{"href":"sugar"}]
}
and you had ingredient resources /Ingredient/apples and
Ingredients/sugar, then you could describe the mapping:
{
"properties":{
"ingredients":{
"baseUrl":"/Ingredient/",
"hrefProperty":"href"
}
}

I think the motivation for this was to try to describe some of Rail's
RESTful data, but I think it falls short of even being able to do that.
Some of the relationship links that are used in Rails require so much
out-of-band information that maybe we shouldn't even attempt the limbo
required to describe it. If we are going to describe some of the
relational mappings in Rails, it is going to require a more in-depth
mechanism.

versionProperty is the other one that was worded incorrectly, right?
This one is included because it I think it is needed for describing
CouchDB data (they use "_rev", I think).

>> "queryEncoding - This property indicates the query encoding format
>>
> that should be used for querying the collection of instances. The query
> can be post-pended to the collection URL to query the collection."
>
> An example would be nice. "the...format" is pretty vague. Is that
> restricted to Query By Example? Or are queryEncoding formats defined in
> the JSON Schema spec?
>
>
>> "If the user wants to update data or create new instances/resources,
>>
> the client knows the format the server expects."
>
> I don't see how that follows from the attributes given.
>

Should say the structure. The structure is defined by the JSON schema
(all the property definitions defined by JSON schema are inherited and
available for the hyper-schemas).

>
>> "When a fragment identifier is a consists solely a simple token, where
>>
> a token is:"
>
> That's almost English ;)
>

Thanks :)

>
>> "token = 1*<any CHAR except CTLs or separators>"
>>
>
> Also the single-quoted-string def. I know you wrapped those in pre tags,
> but the parts between angle brackets aren't being rendered. And the
> smart quotes make it really hard to read :)
>
>
>> "If the reference starts with # instead of object location, the
>>
> current JSON resource should be considered the starting point for the
> property reference."
>
> I think you can just refer to RFC 2396 Sec 5 for all that.
>
>
>> #foo["another prop"]
>>
>
> I think you mean #foo["another%20prop"]. But I don't see why that would
> ever be preferred over #foo.another%20prop...? If the bracket syntax
> were reserved for integer indexing, that would make things much simpler.
>

Bad example. How about:
#foo["has.period"] or #foo["has[bracket"]

That necessitates bracket syntax...


> In general, I like using fragments for this sort of referencing. But I
> worry that we are going to end up implementing Javascript inside the
> schema spec. Or XPath. Either would be extreme IMO.
>

I agree wholeheartedly. I will certainly do all I can to avoid that.

Kris

Kris Zyp

unread,
Jan 2, 2009, 2:49:22 PM1/2/09
to restfu...@googlegroups.com

>
>> #foo["another prop"]
>>
>
> I think you mean #foo["another%20prop"]. But I don't see why that would
> ever be preferred over #foo.another%20prop...? If the bracket syntax
> were reserved for integer indexing, that would make things much simpler.
>
>
What if we just eliminate brackets altogether from the default fragment
identifier resolution mechanism? Integer indexing doesn't really need it
(brackets just help make it look like C-style languages). This seems
like it would make things much easier for implementors. Perhaps we could
escape periods that are part of property names to distinguish between
the property name delimiter:
#foo.has%2Eperiod
Or are escaped characters in URLs supposed to be semantically equivalent
to their corresponding character?
It is would also be tricky to reference an empty string property name.
Perhaps we could just say we can't refer to properties with periods or a
empty string property names

Kris

Robert Brewer

unread,
Jan 2, 2009, 3:50:53 PM1/2/09
to restfu...@googlegroups.com
Kris Zyp wrote:
> >
> >> #foo["another prop"]
> >>
> >
> > I think you mean #foo["another%20prop"]. But I don't see why that
> > would ever be preferred over #foo.another%20prop...? If the bracket
> > syntax were reserved for integer indexing, that would make things
> > much simpler.
> >
> >
> What if we just eliminate brackets altogether from the default
> fragment identifier resolution mechanism? Integer indexing doesn't
> really need it (brackets just help make it look like C-style
> languages). This seems like it would make things much easier
> for implementors. Perhaps we could escape periods that are part
> of property names to distinguish between the property name
> delimiter:
> #foo.has%2Eperiod
> Or are escaped characters in URLs supposed to be semantically
> equivalent to their corresponding character?

That's not strictly defined, but I would bet a lot of URI parsers decode
all components after separating them. Probably best not to risk it.

> It is would also be tricky to reference an empty string property name.
> Perhaps we could just say we can't refer to properties with periods or
> a empty string property names

I'm OK with that.

The only other alternatives I can see would be to adopt an escape
mechanism like "foo.has\.period" or character references like "&#46".
I'm not eager to adopt either of those, but am still thinking over the
implications... How does e.g. XPath handle it?


Robert Brewer
fuma...@aminus.org

Robert Brewer

unread,
Jan 2, 2009, 4:13:15 PM1/2/09
to restfu...@googlegroups.com

That does strike me as odd. You've got a message at /Recipe/apple-pie
and a schema at http://json.com/my-hyper-schema. Data and metadata about
apple-pie go in the former, and data and metadata for the schema go in
the latter. The value of baseUrl, "/Ingredient/", is metadata about
another resource entirely. I personally don't find this style onerous:

GET /Recipe/apple-pie

{
"name":"Apple Pie",

"ingredients":[{"href":"/Ingredients/apples"},{"href":"/Ingredients/suga
r"}]
}

Well, other than the repeated `{"href":` chunks ;)

> versionProperty is the other one that was worded incorrectly, right?
> This one is included because it I think it is needed for describing
> CouchDB data (they use "_rev", I think).

Yes, but is that supposed to be {"versionProperty": "3.0.2"} or
{"versionProperty": "_rev"}? The text is not consistent from one
attribute to another.


Robert Brewer
fuma...@aminus.org

Kris Zyp

unread,
Jan 2, 2009, 4:18:21 PM1/2/09
to restfu...@googlegroups.com

I agree, your representation is certainly superior. However, this was
more of a question of how flexible we want to be in describing existing
data representations. Being able to describe some of Rail's default
representations is what I had in mind, but it does feel like that just
going too far in being accommodating.

I just removed baseUrl.

>> versionProperty is the other one that was worded incorrectly, right?
>> This one is included because it I think it is needed for describing
>> CouchDB data (they use "_rev", I think).
>>
>
> Yes, but is that supposed to be {"versionProperty": "3.0.2"} or
> {"versionProperty": "_rev"}? The text is not consistent from one
> attribute to another.
>

{"versionProperty": "_rev"}

You were correct, it was definitely misworded. I was just trying to give
the rational for it's inclusion in the same paragraph :).

Kris

Reply all
Reply to author
Forward
0 new messages