The Future for JSON Referencing

[Gary Court]

I'd like to ask everyone that has an interest in JSON Schema to please
read and comment on the following:

I'm excited to say that the next version of the JSON Schema
specification (draft-03) is nearing completion! With only a couple
discussions left to finalize on, we can all expect a new draft to be
published soon; and implementations to follow shortly after.

However there is one issue that has come to a stalemate, as only Kris
and I have been discussing it, and it seems as if we can not come to a
consensus. Therefore, I am reaching out to the community to please
provide their comments on the issue of JSON referencing.

I will try to be as impartial as possible in this post, but please
correct me if I have forgotten or got something wrong (Kris).

For the uninformed, JSON Referencing is a method of referencing
another piece of JSON data for cyclical JSON data structures, or reuse
of pre-existing JSON data. In the current draft of JSON Schema
(draft-02), this feature is provided by the hyper-schema spec using
"full" links, and implemented using json-ref's "$ref" syntax
( {"$ref":"uri/to/json"} ).

It was proposed that referencing should be added to the core schema
specification as:
1. It is impossible to bootstrap the core schema without hyper schema
(which is a child of the core schema).
2. It is used often enough that it should be a required core feature.
3. Attributes such as "extends" (a core feature) were added to allow
the reuse of schemas, which are probably defined elsewhere.

Although we agreed that JSON referencing should be a feature that is
clearly specified in the JSON Schema specification, the disagreement
comes from the format that referencing should take. Both formats
specify a URI of a JSON instance (or schema) to be used to replace the
reference at resolution time, but take different forms:

*** Format 1 - Schemas-Can-$ref-Link

If a schema specifies a "$ref" attribute, it should be replaced with
the JSON that has the provided URI.

For example, this is how a schema would use this format for
referencing:

{
"id":"my-card",
"name":"A schema that extends card",
"extends":{"$ref":"http://json-schema.org/card"}
"properties":{
"another-card":{"$ref":"my-card"}
}
}

Advantages:
-Backwards compatable with previous drafts, and compatable with json-
ref.
-If defined in the core schema, it is forward compatable with hyper
schema.

Disadvantages:
-Requires extra characters/noise in the schema.

*** Format 2 - Schemas-Can-Be-URI-Strings

An attribute that accepts a schema should also allow a string value,
where the string value is a URI reference/link to another schema.

For example, this is how a schema would use this format for
referencing:

{
"id":"my-card",
"name":"A schema that extends card",
"extends":"http://json-schema.org/card",
"properties":{
"another-card":"my-card"
}
}

Advantages:
-Schemas are more concise, removes extra characters/noise from the
schema.
-The direct URI is cleaner/more readable.

Disadvantages:
-Type collides with other attributes that already accept a string and
schema object (like "type" or "requires").
-Is a different format then previous draft specifications.
-When searching for references in a schema, the attribute name must be
taken into consideration to determine if the string is a reference.

*** Open for comments

Obviously there may be other ways of accomplishing referencing, and
those ideas are welcome to. However, the sooner we reach a consensus
on this, the sooner the next draft can be published. Please post your
comments to this thread on the JSON Schema group. Existing discussions
on this topic can be found below:

http://groups.google.com/group/json-schema/browse_thread/thread/bbe9aea4a8d1ac43/#4ca6154979f0084d
http://groups.google.com/group/json-schema/browse_thread/thread/5dc7d56a18abd81b/4a87bd2f0e50be30#0f330860dc3619e3

Thank you all!

[mat]

As a third alternative how about this hybrid suggestion which arguably
combines the best (or worst) of the others..

If the value of a string is prefixed with a '$' then the remainder of
the string should be treated as a reference URI

eg..

{
"id":"my-card",
"name":"A schema that extends card",

"extends":"$http://json-schema.org/card",
"properties":{
"another-card":"my-card"
}
}

Advantages..
a) Minimises cruft and noise.
b) Makes it easy to identify references in a schema doc
c) May be used for any attribute value in a schema doc.

Disadvantages..
a) Backwards incompatibilities.

Thanks,
mat

> http://groups.google.com/group/json-schema/browse_thread/thread/bbe9a...http://groups.google.com/group/json-schema/browse_thread/thread/5dc7d...
>
> Thank you all!

[Thomy Berlin]

HI @all,

2010/11/3 Gary Court <gary....@gmail.com>

[...]

 

*** Format 1 - Schemas-Can-$ref-Link

If a schema specifies a "$ref" attribute, it should be replaced with
the JSON that has the provided URI.

For example, this is how a schema would use this format for
referencing:

{
 "id":"my-card",
 "name":"A schema that extends card",
 "extends":{"$ref":"http://json-schema.org/card"}
 "properties":{
   "another-card":{"$ref":"my-card"}
 }
}

Advantages:
-Backwards compatable with previous drafts, and compatable with json-
ref.
-If defined in the core schema, it is forward compatable with hyper
schema.

Disadvantages:
-Requires extra characters/noise in the schema.

  [...]

I pref. format 1 - the 'noise' is not so much as it sounds, and it has another
advantage: it lets the door open for other forms how extends could be reached.
it this point we have only a reference, so there is the attribute '$ref' with the
URI as its value - but how about tomorrow ? if we have a attribute, we can give
it a new name with new value(s) if we need. as we only have a string as in format 2,
it could be harder to implement other ways to deal with extends in the future.

so far from here,
JBScout
 

[Alexandre Morgaut]

Well..; I'd prefer having JSON Schema properties prefix by "$" being considered as links than polluting the value itself

> "$extends":"http://json-schema.org/card",

but I think my prefered solution is the first one:

>> "extends":{"$ref":"http://json-schema.org/card"}

And, sorry for this intrusion, but I'd like to be able to reference an embeded piece of schema with RUL like:

>> "extends":{"$ref":"#schemas.card"}

where the format after the # would be JSON Path like
Note that I may have missed the possibility from another way

> --
> You received this message because you are subscribed to the Google Groups "JSON Schema" group.
> To post to this group, send email to json-...@googlegroups.com.
> To unsubscribe from this group, send email to json-schema...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/json-schema?hl=en.
>

[Chris Miles]

When I first came to JSON Schema I expected to be able to give a
referenced schema a short name or alias and use that alias wherever a
type was required.

The $ref syntax makes it clear that the value is a link, that baseURI
applies, etc.

I've come to appreciate the $ref syntax, 'format 1' gets my +1.

Chris

[Andi]

+1 to 1)

The current way of referencing is good enough, because

1) It is a standard and implemented in some frameworks like dojo.
2) You can unambiguously reference normal values, not only schemas.
3) It is more readable. The author will instantly recognize that it is
a ref.

Doing

{
type:"integer",
title:"otherSchema.title"
}

is ambiguous. Better

{
type:"integer",
title:{$ref: "otherSchema.title"}
}

The hybrid solution isn't unambiguous enough either.

Of course, referencing must be part of the core schema. (The same goes
for "default".)

> >http://groups.google.com/group/json-schema/browse_thread/thread/bbe9a......
>
> > Thank you all!

[Nate Morse]

Is it sufficient to identify a URI by the transport prefix (delimited by "://")?

--Nate

>
> Thank you all!

[Kris Zyp]

On 11/4/2010 5:49 AM, Andi wrote:
> +1 to 1)
>
> The current way of referencing is good enough, because
>
> 1) It is a standard and implemented in some frameworks like dojo.

I'm core committer for Dojo, so it wouldn't be hard to get Dojo updated.

> 2) You can unambiguously reference normal values, not only schemas.

Note that we aren't getting rid of JSON referencing as a possible
referencing mechanism, we just are using it in schemas for the
properties that take schemas. I am not aware of any of these properties
that would take a "normal" value except type and requires. Why would you
create a URL to reference "number", when it is actually shorter to
actually type it?

> 3) It is more readable. The author will instantly recognize that it is
> a ref.
>
> Doing
>
> {
> type:"integer",
> title:"otherSchema.title"
> }
>

Title can't reference a schema, so this wouldn't accept URLs. The
"title" attribute wouldn't introduce any ambiguity.

--
Thanks,
Kris

[monsur]

I also prefer #1. The $ref syntax is clear and allows it to be used
as a drop-in replacement for wherever a schema may go. A third option
I was brainstorming was introducing a "type": "schema" property, such
as:

"another-card": {"type": "schema", "$ref": "mycard"}

So that the referencing syntax matches the JsonSchema syntax. Just a
thought.

Thanks,
Monsur

[Sky Sanders]

Another vote, from me, for #1: "extends": { "$ref": "http://json-schema.org/card" }

[sec_...@gmx.de]

A reference does not need to be a schema. Why not referencing any
value?

[sec_...@gmx.de]

On 4 Nov., 14:36, Kris Zyp <kris...@gmail.com> wrote:
> On 11/4/2010 5:49 AM, Andi wrote:> +1 to 1)
>
> > The current way of referencing is good enough, because
>
> > 1) It is a standard and implemented in some frameworks like dojo.
>
> I'm core committer for Dojo, so it wouldn't be hard to get Dojo updated.
>
> > 2) You can unambiguously reference normal values, not only schemas.
>
> Note that we aren't getting rid of JSON referencing as a possible
> referencing mechanism, we just are using it in schemas for the
> properties that take schemas. I am not aware of any of these properties
> that would take a "normal" value except type and requires. Why would you
> create a URL to reference "number", when it is actually shorter to
> actually type it?> 3) It is more readable. The author will instantly recognize that it is

What if the positive effect of the change you propose, if it creates
ambiguous cases?
Probably I need to reference single values, if I want to use "extends"
with boolean values. Then I only need to change one value, and others
simply inherit. Better than to copy and paste.

> > a ref.
>
> > Doing
>
> > {
> > type:"integer",
> > title:"otherSchema.title"
> > }
>
> Title can't reference a schema, so this wouldn't accept URLs. The
> "title" attribute wouldn't introduce any ambiguity

I think this should bsupported, because it is the standard. schema.js
supports referencing any property with the exception of id. And it
makes sense.
>
> --
> Thanks,
> Kris

[sec_...@gmx.de]

https://github.com/akidee/schema.js/blob/master/schemas/jsonSchemaCore.js#L132

"properties" does not point to a schema, but using a reference here is
useful.

On 4 Nov., 14:36, Kris Zyp <kris...@gmail.com> wrote:

[Andi]

What about the current draft then? It says that you can use URIs
(which includes ids as far as I understand it) instead of schemas. I
think this should be removed.
I either want to be able to reference all properties with $ref OR
being able to reference all properties with a string, which is
ambiguous.

I think it's better to have one clear solution for referencing. In
this case the draft must be changed.

[Dean Landolt]

On Fri, Nov 5, 2010 at 2:34 PM, Andi <andrea...@gmx.de> wrote:

What about the current draft then? It says that you can use URIs
(which includes ids as far as I understand it) instead of schemas. I
think this should be removed.
I either want to be able to reference all properties with $ref OR
being able to reference all properties with a string, which is
ambiguous.

I think it's better to have one clear solution for referencing. In
this case the draft must be changed.

I don't think it's ambiguous at all. No one ever commented on my suggestion that we treat a string as just a relative url and fix a base url that defines all the standard json-schema types. What's wrong with that? It's simple and consistent.

[sec_...@gmx.de]

Then we use both strings (that I can only use to reference schemas)
AND the notation with $ref that can reference simple values.
It is no problem to use both notation in parallel. But it easier to
recognize a reference, if we stick to one notation: the one that is
usable in more cases.

On 5 Nov., 22:35, Dean Landolt <d...@deanlandolt.com> wrote:

[Andi]

That was me.

[Robin Berjon]

On Nov 4, 2010, at 07:13 , Thomy Berlin wrote:
> I pref. format 1 - the 'noise' is not so much as it sounds, and it has another
> advantage: it lets the door open for other forms how extends could be reached.
> it this point we have only a reference, so there is the attribute '$ref' with the
> URI as its value - but how about tomorrow ? if we have a attribute, we can give
> it a new name with new value(s) if we need. as we only have a string as in format 2,
> it could be harder to implement other ways to deal with extends in the future.

+1 from me as well, the syntax overhead is small, it can hardly be construed a big pain.

--
Robin Berjon
Robineko (http://robineko.com/)

[heswell]

I vote for 1. The project with which I am involved makes use of Json
Schemas for complex Json structures. An idiom we use frequently (I'm
not even sure if it complies with
the Json schema spec, we use our own validator) goes like this
(omitting quotes)

properties : {
interval : {$ref : "RelativeTimeInterval", optional:true}
}

..in other words, referencing an existing schema and overriding a
schema attribute. I don't see how we can do this if the schema
reference is a simple string uri


heswell

[Dean Landolt]

On Mon, Nov 8, 2010 at 5:42 PM, heswell <steven...@gmail.com> wrote:

I vote for 1. The project with which I am involved makes use of Json
Schemas for complex Json structures. An idiom we use frequently (I'm
not even sure if it complies with
the Json schema spec, we use our own validator) goes like this
(omitting quotes)

properties : {
  interval : {$ref : "RelativeTimeInterval", optional:true}
}

..in other words, referencing an existing schema and overriding  a
schema attribute. I don't see how we can do this if the schema
reference is a simple string uri

Just to clarify, the alternate proposal was not to remove the possibility to use a $ref.

[heswell]

I see. Personally, I'd still favour the $ref notation for all schema
references. The overhead seems slight and it marks such references
very clearly as references.

On Nov 8, 10:44 pm, Dean Landolt <d...@deanlandolt.com> wrote:

> On Mon, Nov 8, 2010 at 5:42 PM, heswell <steven.he...@gmail.com> wrote:
> > I vote for 1. The project with which I am involved makes use of Json
> > Schemas for complex Json structures. An idiom we use frequently (I'm
> > not even sure if it complies with
> > the Json schema spec, we use our own validator) goes like this
> > (omitting quotes)
>
> > properties : {
> >   interval : {$ref : "RelativeTimeInterval", optional:true}
> > }
>
> > ..in other words, referencing an existing schema and overriding  a
> > schema attribute. I don't see how we can do this if the schema
> > reference is a simple string uri
>

> Just to clarify, the alternate proposal was not to *remove* the possibility
> to use a $ref.

[Gary Court]

Thank you everyone for your input! We are going with format 1 with a
near unanimous vote for it.

[Joe]

Gary,

Sorry to resurrect an old thread but I have a quick question on format
1.

How should a validator differentiate between a $ref value that
contains an id and a $ref value that contains a relative URI?

For instance, taking your example:

{
"id":"my-card",
"name":"A schema that extends card",
"extends":{"$ref":"http://json-schema.org/card"}
"properties":{
"another-card":{"$ref":"my-card"}
}
}

If this schema sits in a directory that also includes another json
schema file called 'my-card', what should the behaviour of the parser
be?

What's the order of precedence? Should an attempt to find a matching
id happen first, then an attempt to use the value as a relative URI
come next?

Cheers

[Gary Court]

The "id" attribute is a URI. Therefore, if this schema has an id of
"my-card", and there is another JSON schema file called "my-card",
then this schema is that file. If not, that is an error as it violates
the URI principle.

Based on your question, I imagine you're writing a validator that will
load a relative file if referenced. Unless you're going to enforce the
condition above, I would recommend checking your internal cache of
schemas first before loading a file. (Faster and probably what the
author is intending)

Other validators (such as JSV) don't load referenced files as it
treats URIs as identifiers and not locations. (Hence URIs, not URLs)

-Gary

[Joe Littlejohn]

Hi Gary,

Thanks a lot for your response. I've re-read the id part of the spec
and your explanation makes total sense.

Cheers

[Joe Littlejohn]

Sorry Gary, one more question:

I notice that the core schema (http://json-schema.org/schema) doesn't
specify additionalProperties for schemas themselves (as far as I can
tell). I assume then that the default (empty) schema applies for
additionalProperties for schemas themselves, and hence a schema may
have additional properties of any type e.g.

{
"id":"my-card",
"name":"A schema that extends card",
"extends":{"$ref":"http://json-schema.org/card"}
"properties":{
"another-card":{"$ref":"my-card"}
}

"myextraproperty":"myextravalue"
}

Is my understanding correct?

Cheers

On 10 January 2011 21:35, Gary Court <gary....@gmail.com> wrote:

[Gary Court]

That's it correct. :-)

-Gary