Relative JSON Pointer / JSON Reference RFC statuses?

146 views
Skip to first unread message

Henry Andrews

unread,
Sep 13, 2016, 11:43:27 PM9/13/16
to JSON Schema
Hi folks,
  Does anyone know anything about the status of the Relative JSON Pointer RFC?  Was any further work done after Geraint published the first draft?  https://tools.ietf.org/html/draft-luff-relative-json-pointer-00

  Both based on its use in existing proposals for future versions of JSON [Hyper-]Schema and from my own experience working with a hyper-schema alternative, relative JSON pointers are a requirement for moving forwards.

  Since they are a fairly simple extension to an already accepted RFC it would seem like that might be a *relatively* easy project to see through.

  And while we're at it, what about the JSON Reference draft?  https://tools.ietf.org/html/draft-pbryan-zyp-json-ref-03 JSON reference is used all over the place, not just in JSON Schema.

  I do not have experience shepherding such things through the IETF, but I am willing to spend time on it if anyone can give me some pointers on how to start.

thanks,
-henry

Matt Palmer

unread,
Sep 14, 2016, 5:39:10 AM9/14/16
to 'Henry Andrews' via JSON Schema
On Tue, Sep 13, 2016 at 08:43:26PM -0700, 'Henry Andrews' via JSON Schema wrote:
> I do not have experience shepherding such things through the IETF, but I
> am willing to spend time on it if anyone can give me some pointers on how
> to start.

Everything you ever wanted to know about the IETF, but had no idea to ask:

https://www.ietf.org/tao.html

- Matt

Austin William Wright

unread,
Sep 15, 2016, 2:24:24 PM9/15/16
to JSON Schema
Are there any standards or drafts that even use 'relative' pointers?

I haven't really seen any compelling cases for them, in any event. "Moving forwards" right now means fixing bad language and broken references, mostly, so I think we can still move forward.

Relative pointers would mean breaking several JSON Schema conventions that are enforced for performance. I think for most all of the new features proposed, there's ways to implement them without relative pointers.

Do you have a list of things that this feature would be desirable for?

Austin.

Henry Andrews

unread,
Sep 15, 2016, 6:51:31 PM9/15/16
to JSON Schema

On Thursday, September 15, 2016 at 11:24:24 AM UTC-7, Austin William Wright wrote:
Are there any standards or drafts that even use 'relative' pointers?

I haven't really seen any compelling cases for them, in any event.

The "advanced templating" proposal, which would now be a v6 proposal, relies on them and was absolutely essential for the success of the JSON Schema-based project at Riverbed.  You can see the template+vars pattern all over the published service definitions.  I can pull up some specific examples if that would be helpful.  We didn't exactly use hyper-schema, but what we did use was mostly assembled out of hyper-schema ideas (I will do another big analysis for hyper-schema soon like the one I did for the re-use proposals).

So to me, hyper-media APIs based on JSON Schema are a non-starter without advanced templating and therefore also relative JSON pointers.    It's simply not feasible without them.
 
"Moving forwards" right now means fixing bad language and broken references, mostly, so I think we can still move forward.

This doesn't have anything to do with moving forwards (or not) on JSON Schema Draft 05 (as purely a clean-up of Draft 04).  I assume you've got the Draft 05 work under control, and while I can help with the various mechanical chores for that, I'm mostly looking ahead to Draft 06.  Draft 05 is a key step in getting this going again, but it (intentionally) will not address the significant limitations of Draft 04.
 
Relative pointers would mean breaking several JSON Schema conventions that are enforced for performance. I think for most all of the new features proposed, there's ways to implement them without relative pointers.

Could you elaborate on this a bit?  Unless I'm forgetting something, if you don't actually use relative pointers in your schema, the performance impact is limited to checking for whether your pointers start with '/' or with a number.  Any code for dealing with relative JSON pointers should only come into play if you find at least one pointer that starts with a number.
 
Do you have a list of things that this feature would be desirable for?

The 'vars' section of a link from within a schema that is re-used in larger schemas.  Using a relative pointer means that the re-usable schema is re-usable from anywhere within a larger schema.  If you must use an absolute pointer, you cannot include links except in the final schema because you will not know what the absolute paths will look like.

I can probably come up with more examples, but that is the most immediately obvious one that cannot be done with an absolute pointer.

thanks,
-henry

Henry Andrews

unread,
Sep 15, 2016, 6:51:53 PM9/15/16
to JSON Schema

Austin William Wright

unread,
Sep 16, 2016, 2:51:28 PM9/16/16
to JSON Schema


On Thursday, September 15, 2016 at 3:51:31 PM UTC-7, Henry Andrews wrote:

On Thursday, September 15, 2016 at 11:24:24 AM UTC-7, Austin William Wright wrote:
Are there any standards or drafts that even use 'relative' pointers?

I haven't really seen any compelling cases for them, in any event.

The "advanced templating" proposal, which would now be a v6 proposal, relies on them and was absolutely essential for the success of the JSON Schema-based project at Riverbed.  You can see the template+vars pattern all over the published service definitions.  I can pull up some specific examples if that would be helpful.  We didn't exactly use hyper-schema, but what we did use was mostly assembled out of hyper-schema ideas (I will do another big analysis for hyper-schema soon like the one I did for the re-use proposals).

So to me, hyper-media APIs based on JSON Schema are a non-starter without advanced templating and therefore also relative JSON pointers.    It's simply not feasible without them.

I think I should take a look at some concrete examples then. I just can't think of anything myself that would need this.

JSON Hyper-schema definitely still needs a bit more functionality to cover even a majority of use cases, I think (for example, see <https://github.com/json-schema-org/json-schema-spec/issues/49>). But I'm really squeamish about turning JSON Schema into a fully fledged Turing-complete language.

Or maybe that is to say, if your JSON API is so compact (or in some cases, so poorly designed) that you need a lot of powerful functionality to extract links from it, maybe we should just define a way to link to a script that will generate a list of links from your JSON document.
 
 
"Moving forwards" right now means fixing bad language and broken references, mostly, so I think we can still move forward.

This doesn't have anything to do with moving forwards (or not) on JSON Schema Draft 05 (as purely a clean-up of Draft 04).  I assume you've got the Draft 05 work under control, and while I can help with the various mechanical chores for that, I'm mostly looking ahead to Draft 06.  Draft 05 is a key step in getting this going again, but it (intentionally) will not address the significant limitations of Draft 04.
 
Relative pointers would mean breaking several JSON Schema conventions that are enforced for performance. I think for most all of the new features proposed, there's ways to implement them without relative pointers.

Could you elaborate on this a bit?  Unless I'm forgetting something, if you don't actually use relative pointers in your schema, the performance impact is limited to checking for whether your pointers start with '/' or with a number.  Any code for dealing with relative JSON pointers should only come into play if you find at least one pointer that starts with a number.

The big theory behind JSON Schema is that if this validates:

var schema = { type:"object", properties:{
    "list": { type: "string" }
} };

assertValid(instance, schema);

// ... then so will this, always, 100% of the time:

assertValid(instance.list, schema.properties.list);

But if you're using relative pointers, then that breaks this property, because the validation function doesn't know what the original "instance" was in this case, only "instance.list".

Or maybe put another way, instances should be self-similar.

 
Do you have a list of things that this feature would be desirable for?

The 'vars' section of a link from within a schema that is re-used in larger schemas.  Using a relative pointer means that the re-usable schema is re-usable from anywhere within a larger schema.  If you must use an absolute pointer, you cannot include links except in the final schema because you will not know what the absolute paths will look like.

I can probably come up with more examples, but that is the most immediately obvious one that cannot be done with an absolute pointer.

I've thought of requesting a "vars" keyword before, because there's some JSON values that you might want to use that would be excessively long or very difficult to reference inside a URI Template. For example, an empty string as a key. (If I wanted to use the value of the empty string named property in {"":"123"} in a URI template, the string <http://example.com/{}> isn't a valid URI template.)

And then while you're over at GitHub, I issued PR <https://github.com/json-schema-org/json-schema-spec/pull/50>, I'd love comments ;)

Thanks,

Austin.
 

thanks,
-henry

Henry Andrews

unread,
Sep 16, 2016, 3:25:25 PM9/16/16
to JSON Schema


On Friday, September 16, 2016 at 11:51:28 AM UTC-7, Austin William Wright wrote:

On Thursday, September 15, 2016 at 3:51:31 PM UTC-7, Henry Andrews wrote:

The "advanced templating" proposal, which would now be a v6 proposal, relies on them and was absolutely essential for the success of the JSON Schema-based project at Riverbed.  You can see the template+vars pattern all over the published service definitions.  I can pull up some specific examples if that would be helpful.  We didn't exactly use hyper-schema, but what we did use was mostly assembled out of hyper-schema ideas (I will do another big analysis for hyper-schema soon like the one I did for the re-use proposals).

So to me, hyper-media APIs based on JSON Schema are a non-starter without advanced templating and therefore also relative JSON pointers.    It's simply not feasible without them.

I think I should take a look at some concrete examples then. I just can't think of anything myself that would need this.

JSON Hyper-schema definitely still needs a bit more functionality to cover even a majority of use cases, I think (for example, see <https://github.com/json-schema-org/json-schema-spec/issues/49>). But I'm really squeamish about turning JSON Schema into a fully fledged Turing-complete language.

Or maybe that is to say, if your JSON API is so compact (or in some cases, so poorly designed) that you need a lot of powerful functionality to extract links from it, maybe we should just define a way to link to a script that will generate a list of links from your JSON document.

I definitely need to dig up some examples.  The concern brought up in issue #49 there is a small fraction of what is needed for hyper-schema to work.

However, I don't know where your "turing-complete language" fears come into play here.  All that is needed is the ability to consistently and clearly map URI template variable names into instance locations.  I will need to more fully describe the hypermedia requirements that I've encountered to cover this, so I'm not going to go through it all in this message.  More soon.

As for other concerns...
  
Relative pointers would mean breaking several JSON Schema conventions that are enforced for performance. I think for most all of the new features proposed, there's ways to implement them without relative pointers.

Could you elaborate on this a bit?  Unless I'm forgetting something, if you don't actually use relative pointers in your schema, the performance impact is limited to checking for whether your pointers start with '/' or with a number.  Any code for dealing with relative JSON pointers should only come into play if you find at least one pointer that starts with a number.

The big theory behind JSON Schema is that if this validates:

var schema = { type:"object", properties:{
    "list": { type: "string" }
} };

assertValid(instance, schema);

// ... then so will this, always, 100% of the time:

assertValid(instance.list, schema.properties.list);

But if you're using relative pointers, then that breaks this property, because the validation function doesn't know what the original "instance" was in this case, only "instance.list".


How are JSON Pointers involved in this at all?  Are we talking about the same thing?  Relative pointers are needed in order to fulfill the same URI template from various data sources (again, I'll address *why* that is needed in a separate email once I pull up some examples).  This doesn't have anything at all to do with instance validation.

I think it is important to consider the various separate goals being addressed by JSON Schema + Hyper-Schema:

* Instance validation (validation properties)
* Hypermedia linking (link description objects and other hyper-schema-defined properties)
* Documentation (annotation properties)

We have the properties broken up by these three purposes, and there should be different considerations and requirements applied to each.  For instance:

* It should be possible to overwrite annotation properties at point of use, but not validation properties
* Sub-properties should validate against the corresponding sub-schema
* Hypermedia properties make use of instance field values in ways that the other properties do not (badly phrased, but hopefully you get my general point here)

If we do want to have principles like "for all instances that validate against a schema, all sub-instances validate against the corresponding sub-schema" then they should be clearly stated in the standard rather than implied by observable behavior.  In particular, I'm not convinced that oneOf satisfies the property that you require, or rather if it does, then it needs to be defined very carefully.  If instance A validates against X but not Y in {oneOf: [X, Y]} then A.foo must validate against X.foo.  However, there is *nothing* we can say about the relationship between A.foo and Y.foo.  It may or may not validate.  A.foo may or may not validate against {oneOf: [X.foo, Y.foo]}, since it may or may not validate against Y.foo.  In other words, you need to know about *how* A validated against {oneOf: [X, Y]} in order to say anything about how A.foo should validate.

thanks,
-henry
Reply all
Reply to author
Forward
0 new messages