Hyper-schema feature proposal: templating for other properties

[Geraint (David)]

OK.  So here's something I've been thinking about for the next version of the hyper-schema spec.  I am currently very keen to have this (or any other proposal with similar functionality) included, as it expands the power of hyper-schema and the range of JSON formats we can describe, with relatively little effort.

It's been bothering me that when you want to include the links in the data, that you have to use a particular form (http://json-schema.org/links).  The whole idea of JSON Schema is that it should not place constraints on the data format.  But currently the link relation value must be in a property called "rel", the link target in a property called "href", the target media type in a property called "mediaType", etc.

This makes JSON Schema incapable of describing JSON formats that include the links in the data, if the link format does not match up exactly with the link format in JSON Hyper-Schema.  It also means that you have to either specify all aspects of the link, or move the entire link definition to the schema.

So I am thinking of enabling templating for other properties than just "href".  This would mean that you could actually use hyper-schemas to describe the link representation you're using - for example, take this non-standard link description:

Data:

{

    "links": [

        {

            "uri": "/html/123",

            "rel": "alternate",

            "type": "text/html"

        }

    ]

}

Hypothetical schema:

{

    "type": "object",

    "properties": {

        "links": {

            "type": "array",

            "items": {"$ref": "#/definitions/customLink"}

        }

    },

    "definitions": {

        "customLink": {

            "links": [

                {

                    "href": {"template": "{uri}"},

                    "rel": {"template": "{rel}"},

                    "mediaType": {"template": "{type}"}

                }

            ]

        }

    }

}

Note - the syntax {"template": "..."} is in no way fixed - the suggestion is simply that templating is enabled at all for other properties.

With this feature, a client would be able to look at the client data, and understand the links specified in "links", even though they do not follow the JSON Schema format.

It also enables us to selectively enhance the links in the data with extra information, according to any constraints we choose.

For example, say that we want to include the links in the client, but we also want to specify that any link with relation "search" should have search parameters provided.  In that case, you could provide an enhanced description of that particular link, providing a fallback for all other links, like this:

{

    "type": "object",

    "properties": {

        "links": {

            "type": "array",

            "items": {"$ref": "#/definitions/customLink"}

        }

    },

    "definitions": {

        "customLink": {

            "description": "Either search link, or the default (if not a search link)",

            "oneOf": [

                {"$ref": "#/definitions/searchLink"},

                {

                    "not": {"$ref": "#/definitions/searchLink"}

                    "allOf": [

                        {"$ref": "#/definitions/otherLink"}

                    ]

                }

            ]

        },

        "searchLink": {

            "properties": {

                "rel": {

                    "enum": ["search"]

                }

            },

            "links": [{

                "href": {"template": "{uri}"},

                "rel": "search",

                "mediaType": {"template": "{type}"},

                "method": "GET",

                "schema": {
                    "title": "Search parameters",

                    "type": "object",

                    "properties": {

                        "searchTerm": {"type": "string"}

                    },

                    "required": ["searchTerm"]

                }

            }]

        },

        "otherLink": {

            "links": [{

                "href": {"template": "{uri}"},

                "rel": {"template": "{rel}"},

                "mediaType": {"template": "{type}"}

            }]

        }

    }

}

In the above case, an entry in "links" with a relation of "search" would be described by the #/definitions/searchLink schema, and any others would be described by the #/definitions/otherLink schema.

So, this feature would allow us to include links in the data, in any form we choose, and also provide extra information for particular links if they are present.

What do people think?

[Geraint (David)]

My apologies - that should be "include the links in the data".
 

[Geraint (David)]

On Wednesday, February 13, 2013 11:00:48 AM UTC, Geraint (David) wrote:

Also, "client data" should be "supplied data". :)

[Philippe Marsteau]

Like it. 

How would you handle the case you receive the following

{

   "prop": "xx",

   "links": [{"rel": "alternate", "uri": "http://xxx/xx.json", "type": "application/json"}, 
               {"rel": "alternate", "uri": "http://xxx/xx.xml", "type": "application/xml"}]

}
in the schema?

[Geraint Luff]

I'm sorry, I don't understand - is the example JSON you gave there the data, and you're asking for the schema?  Or is it a schema?

If it's data - that looks pretty much identical to my original example, just with an extra "prop" property.

If it's the schema - then it's not using the proposed {"template": ...} syntax at all, so it would be exactly the same as current behaviour.

--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Philippe Marsteau]

It is the data, and you initial example was with a "search" (unique link rel). I was basically asking what would a schema look like if you had 2 links relationships, with same "rel" but 2 different "types".

[Geraint Luff]

Oh, right!

Do you need different behaviours for the different media types?  If not, then this would cover you:

{

    "type": "object",

    "properties": {

        "links": {

            "type": "array",

            "items": {"$ref": "#/definitions/customLink"}

        }

    },

    "definitions": {

        "customLink": {

            "links": [

                {

                    "href": {"template": "{uri}"},

                    "rel": {"template": "{rel}"},

                    "mediaType": {"template": "{type}"}

                }

            ]

        }

    }

}

If you do need different behaviour, then this one adds extra information to the XML link:

{
    "type": "object",
    "properties": {
        "links": {
            "type": "array",
            "items": {"$ref": "#/definitions/customLink"}
        }
    },
    "definitions": {
        "customLink": {

            "description": "Either XML link, or the default (if not a XML link)",
            "oneOf": [
                {"$ref": "#/definitions/xmlLink"},
                {
                    "not": {"$ref": "#/definitions/xmlLink"}
                    "allOf": [
                        {"$ref": "#/definitions/otherLink"}
                    ]
                }
            ]
        },
        "xmlLink": {
            "properties": {
                "type": {
                    "enum": ["application/xml"]
                }
            },
            "links": [{
             "title": "XML link"
                "href": {"template": "{+uri}"},
                "rel": "search",
                "mediaType": {"template": "{+type}"},
          ... any other extra info ...
            }]
        },
        "otherLink": {
            "links": [{
                "href": {"template": "{+uri}"},
                "rel": {"template": "{+rel}"},
                "mediaType": {"template": "{+type}"}
            }]
        }
    }
}

[Geraint Luff]

Incidentally, Francis and I are discussing a "switch" syntax in another thread, which would make that last one look slightly tidier:

{
    "type": "object",
    "properties": {
        "links": {
            "type": "array",
            "items": {"$ref": "#/definitions/customLink"}
        }
    },
    "definitions": {
        "customLink": {

            "switch": [
                {
                    "case": {

                        "properties": {
                            "type": {"enum": ["application/xml"]}
                        }
                    },

                    "then": {

                        "links": [{
                             "title": "XML link"
                            "href": {"template": "{+uri}"},
                            "rel": "search",
                            "mediaType": {"template": "{+type}"},
                            ... any other extra info ...
                        }]
                    }
                },

                {
                    "then": {

                        "links": [{
                            "href": {"template": "{+uri}"},

                            "rel": "search",
                            "mediaType": {"template": "{+type}"}

                        }]
                    }
                }
            ]
        }
    }
}

[Philippe Marsteau]

David, could you confirm whether the "customLink" would coexit separately to the current hyperschema LDO type definition? Both types are sharing many aspect, beside "href" being an URL for data vs. an URI-template for LDOs. Should one extend the other? or should these be 2 separate types?

[Geraint Luff]

I'm afraid I don't quite get what you're asking. 

Ideally, actual LDOs would not be used in instances at all - current use of http://json-schema.org/links is kind of a hack.  They are not themselves links, but instead descriptions of how to extract or interpret links from data.  They're "meta-links", really - especially if we enable the {"template": ...} syntax in LDOs, they look and act quite differently.

However, some people might want to represent explicit links in a format that looks reassuringly similar to LDOs (except without the {"template": ...} syntax, obviously).  For that, you could use a schema like this:

{
    "type": "object",
    "properties": {
        "href": {"type": "string"},
        "rel": {"type": "string"},
        "mediaType": {"type": "string"},
        "title": {"type": "string"},
     ...
    },
    "required": ["href", "rel"],
    "links": [{
        "href": {"template": "{+href}"},
        "rel": {"template": "{+rel}"},
        "mediaType": {"template": "{+mediaType}"},
     ...
    }]
}

We would probably provide such a schema on the main site, as http://json-schema.org/data-links or something like that.  So if they're using "LDO-ish" links in their data, they could reference that schema - if not, they can write their own, like "customLink" in the example.

But the two definitions would be separate - LDOs themselves would simply be made powerful enough that they could properly describe links embedded in data.

Does that answer your question at all?

[Philippe Marsteau]

You answered my question. And we are on the same page (for this topic ;)). Basically a LDO is a meta-link, while DL (data-links) is a real link. Just needed your confirmation. 

Let us know when this "data-link" schema will be available so ppl can start making schema reference to it?

[Andrei Neculau]

on-topic: I understand that the format of the links differs here and there, but this won't make the world a better place :) it will just make it more flexible, in a world where some other people are pushing in the order direction to create standards (HAL, Siren, etc). This change btw won't cover the HAL style {links: {someRel: {href: X}}}, will it?

I personally don't see any problem with the current link definition. Might be coincidence, but this is how I coded examples, even before meeting JSON Schema

off-topic: the switch-case makes for a more verbose, and thus less readable format IMO. I would choose the current format (with oneOf or type) any time.

[Francis Galiegue]

On Wed, Feb 20, 2013 at 11:47 PM, Andrei Neculau
<andrei....@gmail.com> wrote:
[...]

>
> off-topic: the switch-case makes for a more verbose, and thus less readable
> format IMO. I would choose the current format (with oneOf or type) any time.
>

Well, I don't know, it looks pretty readable to me. Would you rather have:

{
"oneOf": [ { "$ref": "#/definitions/d1" }, { "$ref": "#/definitions/d2" } ],
"definitions": {
"d1": { "properties": { "p": { "enum": [ "d1" ] } }, "etc": "etc" }
}
}

or

{
"switch": [
{
"when": { "properties": { "p": { "enum": [ "d1" ] } } },
"then": { "etc": "etc" }
}
]
}

(or whatever switch/when/then becomes)?

I prefer to read the second form, but of course YMMV.

--
Francis Galiegue, fgal...@gmail.com
Try out your JSON Schemas: http://json-schema-validator.herokuapp.com

[Andrei Neculau]

I prefer the first one IFF you give it a fair chance and format it properly :)

Thing is it opens the door for more ways to screw up, like

- you start with the switch format

- then you see that you have some definitions that are not "detectable" via "whens"

- so now you have to add also a "oneOf" and reference the "switch" schema, ending up with

{ 

    "oneOf": [ { "$ref": "#/definitions/d1" }, { "$ref": "#/definitions/d2" } ], 
    "definitions": { 

        "d1": { ... },

        "d2": { "switch": .... }
    } 
}

Off-topic: one amendment if I may: add "value" or "constant" definition. Enum does the trick, but it's a semantic workaround IMO, but it's minor. And I do like using one and only one prop that is a list of items, even if the length is 1 (eg. "extends" rather than "extendOne" + "extendMany")

{"enum": ["1"]}

vs

{"value": "1"}

{"const": "1"}

{"constant": "1"}

[Geraint Luff]

On 20 February 2013 22:47, Andrei Neculau <andrei....@gmail.com> wrote:

on-topic: I understand that the format of the links differs here and there, but this won't make the world a better place :) it will just make it more flexible, in a world where some other people are pushing in the order direction to create standards (HAL, Siren, etc). This change btw won't cover the HAL style {links: {someRel: {href: X}}}, will it?

I personally don't see any problem with the current link definition. Might be coincidence, but this is how I coded examples, even before meeting JSON Schema

The thing is that JSON (Hyper-)Schemas are documents that describe data formats - the JSON Schema specification should not affect your data format at all.  Currently, there are hyper-text formats which JSON (Hyper-Schema) cannot describe.  If using JSON Schema to describe your format restricts your format in any way, then we're doing something quite wrong.

Imagine a similar scenario that is not hyper-text, but plain old structural constraints on data.  You have some data format, but somehow the language of JSON Schema is insufficient to describe it.  The answer at that point should not be for you to change your data format so that JSON Schema can describe it - the answer is that JSON Schema is missing some required functionality.

So I find it extremely difficult to accept any solution that says "HAL does things one way, Siren another, and JSON Schema yet another".  JSON Schema is not a data format!  It is a language for describing data formats, and if there are formats we cannot describe then that is a problem.

Also, you are right - this proposal would not cover HAL-style links, and thus is incomplete.  I would welcome suggestions for an alternative syntax that could cover that case as well.

[Andrei Neculau]

Request to hijack this thread. A birdie whispers "Granted", and the shit hits the fan.

I think this tries to solve smth on a lower-level than the level of the problem itself.

A bit of my work context - a JSON schema describes a media-type. That media-type may be used by 5 resources (please slap me if I'm RESTish and I lose focus). Each resource may link to diff other resources, plus each resource may or may not show the links, depending on state, authorization, etc

Given that, in my current context I should not have any links in JSON Schemas. I do it though via a generator by collecting all possible links that those 5 resources can have, and putting those in all media-types that those resources can respond with. This is mainly for spec-ing (makes it visible what is possible to implement) and for documentation convenience (if you get X, then that X could maybe give links to Y, Z,..).

So, after the whole blob above, what I'm trying to say is that one should maybe possibly... :) [mind me, I'm really not trying to start a fire] NOT have links in the JSON Schema describing the instance.

1) If I get some JSON instance, why assume that I can now go to another JSON instance?

2) If the JSON schema describes a representation, then why would 2 representations of the same resource have different links? which can easily happen

Coming back to the original thread

=========================

It only now hit me that you initially wrote:

"This makes JSON Schema incapable of describing JSON formats that include the links in the data, if the link format does not match up exactly with the link format in JSON Hyper-Schema.  It also means that you have to either specify all aspects of the link, or move the entire link definition to the schema."

In what way does JSON Schema restrict my format, may it be HAL, Siren, etc ?

I believe I am very much able to describe any format I want. {"type": "object", "properties": {...anything goes here..}}

Indeed, with the overhead of

1) either linking to elements of the http://siren.com/links schema

2) either specifying myself how the link structure looks like

but that overhead is because of flexibility.

{"properties": {"links": {"$ref": "https://json-schema.org/hyperschema"}}}

vs

{"properties": {"links": {"properties":  {"uri": {"$ref": "https://json-schema.org/hyperschema#/properties/href"}, "relation": {"$ref": "https://json-schema.org/hyperschema#/properties/href"} }}}

vs

{"id": "customhyperschema", ...}

{"properties": {"links": {"$ref": "customhyperschema"}}}

Am I off base here?

[frontend_dev]

Off-topic: one amendment if I may: add "value" or "constant" definition. Enum does the trick, but it's a semantic workaround IMO, but it's minor. And I do like using one and only one prop that is a list of items, even if the length is 1 (eg. "extends" rather than "extendOne" + "extendMany")

{"enum": ["1"]}

vs

{"value": "1"}

{"const": "1"}

{"constant": "1"}

Yes, I would like this, too.

[Geraint Luff]

On 21 February 2013 14:06, Andrei Neculau <andrei....@gmail.com> wrote:

Request to hijack this thread. A birdie whispers "Granted", and the shit hits the fan.

I think this tries to solve smth on a lower-level than the level of the problem itself.

A bit of my work context - a JSON schema describes a media-type. That media-type may be used by 5 resources (please slap me if I'm RESTish and I lose focus). Each resource may link to diff other resources, plus each resource may or may not show the links, depending on state, authorization, etc

Given that, in my current context I should not have any links in JSON Schemas. I do it though via a generator by collecting all possible links that those 5 resources can have, and putting those in all media-types that those resources can respond with. This is mainly for spec-ing (makes it visible what is possible to implement) and for documentation convenience (if you get X, then that X could maybe give links to Y, Z,..).

So, after the whole blob above, what I'm trying to say is that one should maybe possibly... :) [mind me, I'm really not trying to start a fire] NOT have links in the JSON Schema describing the instance.

1) If I get some JSON instance, why assume that I can now go to another JSON instance?

2) If the JSON schema describes a representation, then why would 2 representations of the same resource have different links? which can easily happen

OK - I really want to clarify something here:  are you talking about actual links, or link descriptions?

Actual links (as in, the actual URIs) should in general not be in the schema.  What the schema holds is a description of where to find links in the data.  The actual link itself is not in the schema.

I just wanted to be clear on what exactly we're discussing here.

Coming back to the original thread

=========================

It only now hit me that you initially wrote:

"This makes JSON Schema incapable of describing JSON formats that include the links in the data, if the link format does not match up exactly with the link format in JSON Hyper-Schema.  It also means that you have to either specify all aspects of the link, or move the entire link definition to the schema."

In what way does JSON Schema restrict my format, may it be HAL, Siren, etc ?

I believe I am very much able to describe any format I want. {"type": "object", "properties": {...anything goes here..}}

Indeed, with the overhead of

1) either linking to elements of the http://siren.com/links schema

Right - what does the schema at that URI actually contain?

Currently, it can't contain any definitions for how to extract various components of the hyper-link from the format.  All it can contain is a documentation of what properties can exist.

So, your format can use Siren-style links in the data, and it can declare this in the schema by linking to the http://siren.com/links schema.  But in order for the client to be able to use those links, the client must have special behaviour hard-wired into it for any data following that particular schema.

That's exactly what I'm trying to get away from - this hard-wiring of the clients to recognise certain link formats based on the URIs of their schemas.  I would like a client that has never seen a Siren link before to be able to fetch "http://siren.com/links", and then suddenly understand how to interpret the links, instead of requiring special knowledge to be baked into it by a programmer.
 

2) either specifying myself how the link structure looks like

but that overhead is because of flexibility.

{"properties": {"links": {"$ref": "https://json-schema.org/hyperschema"}}}

vs

{"properties": {"links": {"properties":  {"uri": {"$ref": "https://json-schema.org/hyperschema#/properties/href"}, "relation": {"$ref": "https://json-schema.org/hyperschema#/properties/href"} }}}

vs

{"id": "customhyperschema", ...}

{"properties": {"links": {"$ref": "customhyperschema"}}}

Am I off base here?

On Thursday, February 21, 2013 12:31:48 PM UTC+1, Geraint (David) wrote:

On 20 February 2013 22:47, Andrei Neculau <andrei....@gmail.com> wrote:

on-topic: I understand that the format of the links differs here and there, but this won't make the world a better place :) it will just make it more flexible, in a world where some other people are pushing in the order direction to create standards (HAL, Siren, etc). This change btw won't cover the HAL style {links: {someRel: {href: X}}}, will it?

I personally don't see any problem with the current link definition. Might be coincidence, but this is how I coded examples, even before meeting JSON Schema

The thing is that JSON (Hyper-)Schemas are documents that describe data formats - the JSON Schema specification should not affect your data format at all.  Currently, there are hyper-text formats which JSON (Hyper-Schema) cannot describe.  If using JSON Schema to describe your format restricts your format in any way, then we're doing something quite wrong.

Imagine a similar scenario that is not hyper-text, but plain old structural constraints on data.  You have some data format, but somehow the language of JSON Schema is insufficient to describe it.  The answer at that point should not be for you to change your data format so that JSON Schema can describe it - the answer is that JSON Schema is missing some required functionality.

So I find it extremely difficult to accept any solution that says "HAL does things one way, Siren another, and JSON Schema yet another".  JSON Schema is not a data format!  It is a language for describing data formats, and if there are formats we cannot describe then that is a problem.

Also, you are right - this proposal would not cover HAL-style links, and thus is incomplete.  I would welcome suggestions for an alternative syntax that could cover that case as well.

[Andrei Neculau]

OK - I really want to clarify something here:  are you talking about actual links, or link descriptions?
Actual links (as in, the actual URIs) should in general not be in the schema.  What the schema holds is a description of where to find links in the data.  The actual link itself is not in the schema.

Irrelevant to my example IMO, but link descriptions. Irrelevant thought because you cannot describe links (to an API client) prior to asses state, authorization etc (in some cases).

I would rephrase smth that you said though: "What the schema holds is a description of how to find spec-ed links and how to build their URIs from the data". Like the example on Page 3 http://tools.ietf.org/html/draft-luff-json-hyper-schema-00 ---- by looking at the schema of a Written Article, I know that an instance of a Written Article will link to "full" and "author" via {id} and /user?id={authorId}.

Right?

Coming back - I'm starting to get a grip of what you're trying to do. Give me a chance to phrase it.

In the example you give in your first post, you're actually telling the "json schema reader": hey, each item in "links" actually has 1 link with href pointing to the value of the "uri" key, and with mediaType pointing to the value of "type".

Off base?

If I'm correct, then what I don't get is what is the difference between

"href": {"template": "uri"}

and

"href": "{uri}"

?

Granted there is still issues - only the URI can be a "URI template", so you cannot do today "mediaType": "{type}"..... talking out loud here.

Thanks for the patience! It's a nice idea (you already know that :) ), but I wonder how much do you get out of it.

I'm speaking from the top of my head.. but there is basically the HAL style, and the "say JSON Schema" style (some have links, some _links), and nested or only first-level links. I hope there's not going to be more variations without arguments for breaking out.

[Geraint Luff]

On 21 February 2013 22:49, Andrei Neculau <andrei....@gmail.com> wrote:

OK - I really want to clarify something here:  are you talking about actual links, or link descriptions?
Actual links (as in, the actual URIs) should in general not be in the schema.  What the schema holds is a description of where to find links in the data.  The actual link itself is not in the schema.

Irrelevant to my example IMO, but link descriptions. Irrelevant thought because you cannot describe links (to an API client) prior to asses state, authorization etc (in some cases).

I would rephrase smth that you said though: "What the schema holds is a description of how to find spec-ed links and how to build their URIs from the data". Like the example on Page 3 http://tools.ietf.org/html/draft-luff-json-hyper-schema-00 ---- by looking at the schema of a Written Article, I know that an instance of a Written Article will link to "full" and "author" via {id} and /user?id={authorId}.

Right?

Coming back - I'm starting to get a grip of what you're trying to do. Give me a chance to phrase it.

In the example you give in your first post, you're actually telling the "json schema reader": hey, each item in "links" actually has 1 link with href pointing to the value of the "uri" key, and with mediaType pointing to the value of "type".

Off base?

Spot on.

 

If I'm correct, then what I don't get is what is the difference between

"href": {"template": "uri"}

and

"href": "{uri}"

?

There is no difference - just a syntax for illustration.  I was using the {"template": ...} syntax for other properties (such as "mediaType" or "rel") to accentuate the fact that they were templated.
 

Granted there is still issues - only the URI can be a "URI template", so you cannot do today "mediaType": "{type}"..... talking out loud here.

Precisely: that was the purpose of the {"template": ...} syntax - it maintained backwards-compatability with the current version (no accidental templating), and was clear to the reader whether the value was intended to be a template or not.

The syntax is not something I was fixed on - I would be equally happy enabling "mediaType": "{type}".  I'm sorry if it was confusing... :S

 

Thanks for the patience! It's a nice idea (you already know that :) ), but I wonder how much do you get out of it.

I'm speaking from the top of my head.. but there is basically the HAL style, and the "say JSON Schema" style (some have links, some _links), and nested or only first-level links. I hope there's not going to be more variations without arguments for breaking out.

 

I'm already uncomfortable with using the "say JSON Schema" style - I dislike the client having to look for the special-case schemas "http://json-schema.org/links" or "http://stateless.co/hal-links".

I want to be able to say "if your client understands JSON Hyper-Schema, then you can crawl or use unfamiliar APIs".  If I have to say "if your client understands JSON Hyper-Schema, and has this set of rules hard-wired for this schema, and this other set for this other schema, ...", then I've lost something.  If we can (as a format) describe other JSON hyper-text formats (which I think is what a "hyper-schema" should mean) then we can sell ourselves as the only format a client needs to support to do all sorts of cool stuff (like crawling an unfamiliar API, for instance).

Many of the variants that look mostly like JSON Schema -style links differ slightly (most often "mediaType" => "type").  I could change the JSON Schema link format to follow the current fashion, but I'd rather have a more flexible definition that can account for the differences.