Proposal: merge-patch support for draft v5

[Francis Galiegue]

This would be an _optional_ feature.

Remember that at one time I also proposed JSON Patch support (RFC
6901)? Well, a new draft is there which, while far less powerful than
JSON Patch, makes it easier and plenty enough in the case of JSON
Schema since it applies to JSON objects only: json-merge-patch.

Draft here:

http://tools.ietf.org/html/draft-ietf-appsawg-json-merge-patch-00

Basically, what I proposed was:

{
"patch": {
"schema": { "some": "schema", "note": "could be a $ref" },
"with": [ "json", "patch", "here" ]
}
}

Well, I propose this:

{
"merge": {
"schema": { "some": "schema" },
"with": { "json": [ "merge-patch", "here" ] }
}
}

The rules would be:

* it is second in priority only to $ref;
* like with $ref, if it is found, all other keywords are ignored;
* the result of the substitution is the merged schema.

So, for instance:

{
"merge": {
"schema": {
"type": "object",
"properties": {
"p": { "type", "integer" }
},
"required": [ "p" ]
},
"with": {
"required": [ "q" ],
"properties": {
"p": null,
"q": { "type": "array" }
}
}
}
}


would give, once processed:

{
"type": "object",
"required": [ "q" ],
"properties": {
"q": { "type": "array" }
}
}


We could even add JSON Patch support this way. The order would then be
$ref, patch, merge.

Comments?
--
Francis Galiegue, fgal...@gmail.com
JSON Schema in Java: http://json-schema-validator.herokuapp.com

[Geraint]

This is to get around the situation where you want to mark something as not extendable (additionalProperties: false) but you still want to extend it, yes?

As before, I'm against it on the principle that it's overly complicated - and I don't think that making it optional would actually help at all.

Let's contrast it with "format": "format" is optional, but schema authors are strongly advised to write their schemas on the assumption that "format" is not supported by the target validator - if the validator does support it, it's a small bonus, but it should not be relied upon.

However, how would that work with this proposal?  I can't see a way to write a schema that fails gracefully if this feature is not supported - so with this proposal you would end up with a class of schemas that become useless when you take them out of the specific environment they were written for.

If a feature is "optional", then schemas written with it must be able to degrade gracefully on tools that do not support it.  Otherwise, it's not really an optional feature, it's a non-backwards-compatible extension to the standard.

[Francis Galiegue]

On Wed, Jun 26, 2013 at 5:05 PM, Geraint <gerai...@gmail.com> wrote:
> This is to get around the situation where you want to mark something as not
> extendable (additionalProperties: false) but you still want to extend it,
> yes?
>
> As before, I'm against it on the principle that it's overly complicated -
> and I don't think that making it optional would actually help at all.
>

Let's face it: THE DEMAND IS THERE. And JSON Schema just has no
support for it at the moment.

As to "overly complicated", uhm. Read the draft ;) It is, on the
opposite, _very_ simple.

If it is the "optional" part which troubles you we can make it
compulsory instead... In fact, that is a better bet.

Now, what do other users think is the question. But I believe this
will be a HUGE bonus to JSON Schema.

[Francis Galiegue]

On Wed, Jun 26, 2013 at 5:12 PM, Francis Galiegue <fgal...@gmail.com> wrote:
[...]

>
> As to "overly complicated", uhm. Read the draft ;) It is, on the
> opposite, _very_ simple.
>

Let me enhance that fact again: it is SIMPLE.

The argument against JSON Patch being complicated was relevant (but
after all, I could achieve it and I am no genius), but this argument
falls down the toilet with json-patch-merge.

[Geraint]

On Wednesday, June 26, 2013 4:12:54 PM UTC+1, fge wrote:

On Wed, Jun 26, 2013 at 5:05 PM, Geraint <gerai...@gmail.com> wrote:
> This is to get around the situation where you want to mark something as not
> extendable (additionalProperties: false) but you still want to extend it,
> yes?
>
> As before, I'm against it on the principle that it's overly complicated -
> and I don't think that making it optional would actually help at all.
>

Let's face it: THE DEMAND IS THERE. And JSON Schema just has no
support for it at the moment.

As to "overly complicated", uhm. Read the draft ;) It is, on the
opposite, _very_ simple.

If it is the "optional" part which troubles you we can make it
compulsory instead... In fact, that is a better bet.

Now, what do other users think is the question. But I believe this
will be a HUGE bonus to JSON Schema.

Whoah, capitals!  :)

If I understand correctly, the demand is there because people wish to override "additionalProperties":false, because they wish to ban "unknown" properties (for input to object-loaders in strictly-typed languages) but they then want to extend their previous schemas.

However, I feel that JSON Patch-Merge is an extremely heavyweight solution to this problem.  It's not just implementing the merge (which I agree is fairly simple) - it's that:

• we require an extra pre-processing step - more processing time, more memory (the result of each merge is an independent copy)
  
• the resuting JSON Schema produced by the merge has no definite URI - if it's a composite of two schemas from separate places in the document, what's the JSON Pointer fragment path?

The second one of those is the one that concerns me most.  We will be dealing with a portion of a schema that (in terms of the actual document) does not really exist.  Providing answers to simple questions like "where did the validation fail, and why?" suddenly become complicated to answer.

[Geraint]

What I would prefer is a solution that more closely targeted the problem at hand.

Have we considered something like an "unknownProperties" option (boolean only) that could be used by these schemas for strictly-typed data-loaders (instead of "additionalProperties")?

[Francis Galiegue]

On Wed, Jun 26, 2013 at 5:32 PM, Geraint <gerai...@gmail.com> wrote:
[...]
>
> Whoah, capitals! :)
>

Sorry for that ;)

>
> If I understand correctly, the demand is there because people wish to
> override "additionalProperties":false, because they wish to ban "unknown"
> properties (for input to object-loaders in strictly-typed languages) but
> they then want to extend their previous schemas.
>

That is one scenario. But there are others.

Suppose you wish to reuse an existing schema, but want to add some
more properties to it, or alter the definition a little: refer to it,
patch it. As to...

> However, I feel that JSON Patch-Merge is an extremely heavyweight solution
> to this problem. It's not just implementing the merge (which I agree is
> fairly simple) - it's that:
>
> we require an extra pre-processing step - more processing time, more memory
> (the result of each merge is an independent copy)

Only if "merge" is there!

> the resuting JSON Schema produced by the merge has no definite URI - if it's
> a composite of two schemas from separate places in the document, what's the
> JSON Pointer fragment path?
>

To this, the answer is: "schema authors modifying external schemas by
the means of "merge" MUST ensure that the URI defining the location of
this schema, if any, differs from the original".

Basically, there should always be an "id" (by the way, when do we get
rid of fragment-only "id"s) in the merged schema if the original
schema has one.

[Geraint]

On Wednesday, June 26, 2013 4:39:34 PM UTC+1, fge wrote:

On Wed, Jun 26, 2013 at 5:32 PM, Geraint <gerai...@gmail.com> wrote:
[...]
>
> Whoah, capitals!  :)
>

Sorry for that ;)

>
> If I understand correctly, the demand is there because people wish to
> override "additionalProperties":false, because they wish to ban "unknown"
> properties (for input to object-loaders in strictly-typed languages) but
> they then want to extend their previous schemas.
>

That is one scenario. But there are others.

Suppose you wish to reuse an existing schema, but want to add some
more properties to it, or alter the definition a little: refer to it,
patch it. As to...

That's not a use-case, that's a description of the feature.  If you asked "Why would anyone need a validator?" and I answered "So they can validate things!", that's not really an answer. :p  You were probably hoping for something like "For automated testing, to see if the service is producing the right shape of data".

What situations might require someone to want to patch a schema - in a way that is not covered by proper schema inheritance using "allOf"?

Overriding "additionalProperties":false is the only one I can remember off the top of my head, and I think there's a different underlying issue that can be addressed for that one.

 

> However, I feel that JSON Patch-Merge is an extremely heavyweight solution
> to this problem.  It's not just implementing the merge (which I agree is
> fairly simple) - it's that:
>
> we require an extra pre-processing step - more processing time, more memory
> (the result of each merge is an independent copy)

Only if "merge" is there!

Well, you still require the tools to be able to perform the pre-processing, and work off merged copies of schemas.

I take your point, though - it complicates the design of simple tools by a relatively small amount.

 

> the resuting JSON Schema produced by the merge has no definite URI - if it's
> a composite of two schemas from separate places in the document, what's the
> JSON Pointer fragment path?
>

To this, the answer is: "schema authors modifying external schemas by
the means of "merge" MUST ensure that the URI defining the location of
this schema, if any, differs from the original".

Basically, there should always be an "id" (by the way, when do we get
rid of fragment-only "id"s) in the merged schema if the original
schema has one.

Just differing from the original is not my concern.

Say we have some data at URI "/data/myData", and a schema at "/schemas/mySchema".  Due to the context-free nature of schema vaildation, an automated tool processing that data can produce an output like:

{
    "valid": "false",
    "firstError": {
        "description": "The value is of the wrong type - must be an integer",
        "schemaKeyword": "type",
        "dataWhere": "/data/myData#/id",
        "schemaWhere": "/schemas/mySchema#/properties/id"
    }
}

For any given validation error, it is possible to hone in on the exact part of the data that is the issue, and the exact schema that caused it to fail.  The context-free part means that that "honed-in" validation should still produce the same error - however, honing-in like that only makes sense if the schema in question actually exists.

A merged schema will exist, and it can be given a URI like "/schemas/mySchema#merge-result:1542", but that URI is not resolvable - it's simply an internal (possibly temporary) identifier used by that particular validator.

[Francis Galiegue]

On Wed, Jun 26, 2013 at 6:05 PM, Geraint <gerai...@gmail.com> wrote:
[...]
>

> That's not a use-case, that's a description of the feature. If you asked
> "Why would anyone need a validator?" and I answered "So they can validate
> things!", that's not really an answer. :p You were probably hoping for
> something like "For automated testing, to see if the service is producing
> the right shape of data".
>
> What situations might require someone to want to patch a schema - in a way
> that is not covered by proper schema inheritance using "allOf"?
>

I've had a situation just like this: one issue I have opened was a
misinterpretation of the schema where what the author wanted to do was
in fact patch quite a large schema (more than 240 lines) so that _one_
property changed definition.

So, rather than split the schema again and again, a simple patch-merge
will do the trick. That is one example usage.

And I disagree with your "that's a description of the feature".
Reusing existing schemas is an essential part of JSON Schema. Only
today, what is really only available is $ref, and it only adds, never
modifies. This proposal allows seamless modifications.

[...]

Eh, unresolvable URIs are plenty enough, I honestly don't think of it
as a concern at all ;)

Which brings another point: addressing still needs to be fixed in a
definitive, non ambiguous manner. We are still far from being there.

[Lee Whitney]

My first impression after reading this proposal was that it is overly complex.

I tried to read the corresponding ietf draft document that describes the use cases but can't find it.

Does anyone have a link to it?