$patch and $merge, an explanation -- plus strictProperties

[Francis Galiegue]

OK, so, first of all:

FORGET ABOUT INHERITANCE. This is a purely practical problem.

Problem: I have a schema which defined properties, and doesn't allow
any more properties than what is defined; I want to allow a new one.
How do I do?

Solution:
{
"$merge": {
"source": { /* original schema here or a $ref to it */ },
"patch": {
"properties": { "newProperty": { /* schema for new
property here */ } }
}
}
}

Problem: I have a list of values in an enum in a given schema but I
want to add one. How do I do?

Solution:
{
"$patch": {
"source": { /* original schema here or a $ref to it */ },
"patch": [
{ "op": "add", "path": "/enum/-", "value": /* new enum value */ }
]
}
}

There. Simple, to the point, uses two existing RFCs, DOES NOT BREAK
BACKWARDS COMPATIBILITY.

Also, strictProperties: this is a proposal I made to also ease the
difficulty of writing schemas for objects with additional properties
forbidden etc, plus required; it is a boolean and its definition is
simple enough. When it is false, no effect. When it is true:

* all properties defined in properties MUST be present;
* all properties not defined in either properties,
additionalProperties or patternProperties MUST NOT be present.

Again, simple and to the point!

--
Francis Galiegue, fgal...@gmail.com, https://github.com/fge
JSON Schema in Java: http://json-schema-validator.herokuapp.com
Parsers in pure Java: https://github.com/parboiled1/grappa (redde
Caesaris: https://github.com/sirthias)

[Sebastian Lasse]

Hey Francis,

thanks for the explanation + I would "vote" for these as well.

But I doubt that the google group is the right place for new proposals as the discussion about v5 becomes pretty defragmented here.

At least we would need a unique tag for that here. You won't find it here if you search for e.g. v5 proposals...

Did you notice that there is a special page in the official json-schema repo for v5 proposals? It is a Wiki page that anyone can edit.

I already put your "translations" syntax there (I have credited/blamed you there ;) ) - but I sent you a mail before (from "redaktor").

In cc. to Geraint an Kris - but nobody noticed ;))

It makes sense for me to bundle all proposals in this github pages (45 so far).

Geraint Luff already created this and some more about merging.

strictProperties sounds as if it might be related to this.

Once the proposals from here are transferred from github we (the before mentioned and each one interested) should focus on an active discussion on v5

together with Kris Zyp who is the inventor of this (met the chief of sitepen before, but unfortunately not him too). Kris must be incredible active with dstore

and some other dojo future things as you can see on github ;) But that is ok because I am using and liking dstore as well :)

I would also be totally openminded about creating an (inofficial) v5 meta-schema which would be a "WIP - may change anytime".

btw: I am also in favour of $data and $propertyLinks.

Most important:

What I would expect from v5 is that it closes some gaps between the semantically XML web and the semantically JSON web.

That means basically XML schema to JSON schema and RDFS/OWL XML to JSON hyperschema.

Your proposals and the mentioned ones basically would do that.

Additionally: 

• All the "predefined XML schema datatypes" should go to "format" suggestions. 
  

I defined this schemas already. See my [announce] owl2jsonschema.js post probably coming tomorrow (23:50 now ;)

• There should be a special indication (apart from format) saying that this JSON schema relates to an OWL reference or XML schema.

[Austin Wright]

Why forget about inheritance? It sounds like you're arguing against someone. :p

Nonetheless, relationships between instances and types are a very important concern for e.g. JSON document databases. Merge/patch preserves exactly zero data semantics: If I want to say "store properties inherited from base in the `base` table, store subclass-specific properties in `items` table", this doesn't let you do anything like that.

Suppose I want to say "Only validate one property of this instance using that schema over there" - merge/patch not only loses all semantic meaning, but it's simply incapable of solving this problem altogether.

Re: "DOES NOT BREAK BACKWARDS COMPATIBILITY," my v4-compliant library won't parse this correctly. I'd call that backwards-incompatible.

This seems like a fancy way to clone a schema and make small changes. Idk why you'd want to use this instead of copy-and-paste, unless you need to save disk space. At which point you could just store a diff, which is probably even more efficient, space-wise.

[Francis Galiegue]

On Sat, Jan 17, 2015 at 12:51 AM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
> Why forget about inheritance? It sounds like you're arguing against someone.
> :p
>
> Nonetheless, relationships between instances and types are a very important
> concern for e.g. JSON document databases. Merge/patch preserves exactly zero
> data semantics: If I want to say "store properties inherited from base in
> the `base` table, store subclass-specific properties in `items` table", this
> doesn't let you do anything like that.
>

JSON Schema is not an ORM description language... Also, you will
actually never process such a schema since implementations would be
required to expand before processing the schema.

Also, JSON Schema is not meant to validate only JSON Objects. You are
yet another person who seems to be forgetting that...

> Suppose I want to say "Only validate one property of this instance using
> that schema over there" - merge/patch not only loses all semantic meaning,
> but it's simply incapable of solving this problem altogether.
>

Good -- since JSON Schema is not meant for such things.

> Re: "DOES NOT BREAK BACKWARDS COMPATIBILITY," my v4-compliant library won't
> parse this correctly. I'd call that backwards-incompatible.
>

Let me guess: your parser outputs an error on unknown keywords in
schemas... If it does so it isn't compliant!

> This seems like a fancy way to clone a schema and make small changes. Idk
> why you'd want to use this instead of copy-and-paste, unless you need to
> save disk space. At which point you could just store a diff, which is
> probably even more efficient, space-wise.
>

It is just meant to allow schema reuse in a MUCH more efficient manner
than anything existing. Without breaking backwards compatibility,
basing itself on standards etc.

Also, if you use RFC 6902, you even have a "test" operation; which
means you can make expansion fail if some field in the source schema
you want to extend isn't what you expect it to be anymore!

[Austin Wright]

On Friday, January 16, 2015 at 5:06:27 PM UTC-7, fge wrote:

On Sat, Jan 17, 2015 at 12:51 AM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
> Why forget about inheritance? It sounds like you're arguing against someone.
> :p
>
> Nonetheless, relationships between instances and types are a very important
> concern for e.g. JSON document databases. Merge/patch preserves exactly zero
> data semantics: If I want to say "store properties inherited from base in
> the `base` table, store subclass-specific properties in `items` table", this
> doesn't let you do anything like that.
>

JSON Schema is not an ORM description language... Also, you will
actually never process such a schema since implementations would be
required to expand before processing the schema.

That's exactly what it is. It's describing the structure of data.

Some people use it to validate instances

Some people use it to extract link relations

Some people use it to generate forms

Some people use it to structure data storage.

 

Also, JSON Schema is not meant to validate only JSON Objects. You are
yet another person who seems to be forgetting that...

What gives you this impression? I said JSON instances, and I'm so completely not concerned about validation I didn't even mention it.

 

> Suppose I want to say "Only validate one property of this instance using
> that schema over there" - merge/patch not only loses all semantic meaning,
> but it's simply incapable of solving this problem altogether.
>

Good -- since JSON Schema is not meant for such things.

Yes it is. Quoth draft-4:

JSON Schema defines the media type "application/schema+json", a JSON based format for defining the structure of JSON data. JSON Schema provides a contract for what JSON data is required for a given application and how to interact with it. JSON Schema is intended to define validation, documentation, hyperlink navigation, and interaction control of JSON data.

I'd think that pretty clearly covers the case I posted about.

> Re: "DOES NOT BREAK BACKWARDS COMPATIBILITY," my v4-compliant library won't
> parse this correctly. I'd call that backwards-incompatible.
>

Let me guess: your parser outputs an error on unknown keywords in
schemas... If it does so it isn't compliant!

That's an option, but that's not enabled by default.

Without the strict mode option, it will produce false negatives and false positives.

Either way, it's a failure.

 

> This seems like a fancy way to clone a schema and make small changes. Idk
> why you'd want to use this instead of copy-and-paste, unless you need to
> save disk space. At which point you could just store a diff, which is
> probably even more efficient, space-wise.
>

It is just meant to allow schema reuse in a MUCH more efficient manner
than anything existing. Without breaking backwards compatibility,
basing itself on standards etc.

Also, if you use RFC 6902, you even have a "test" operation; which
means you can make expansion fail if some field in the source schema
you want to extend isn't what you expect it to be anymore!

Your $patch/$merge is currently defined as a preprocessing step, before it hits the validator proper (including "test"). If you're proposing a change to these semantics, could you define it please?

Austin. 

[Sebastian Lasse]

Why forget about inheritance? It sounds like you're arguing against someone. :p

fge: Did not got that as well (?)

Also, you will actually never process such a schema since implementations would be 
required to expand before processing the schema. 

Austin: I think this is the (only) important point here ;)

Your $patch/$merge is currently defined as a preprocessing step, before it hits the validator proper (including "test"). If you're proposing a change to these semantics, could you define it please?

fge:  I think this is the (only) important point here ;)

[Francis Galiegue]

On Sat, Jan 17, 2015 at 6:43 AM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...@

>
>>
>> JSON Schema is not an ORM description language... Also, you will
>> actually never process such a schema since implementations would be
>> required to expand before processing the schema.
>
>
> That's exactly what it is. It's describing the structure of data.
>

No, it isn't. Look again.

> Some people use it to validate instances
>

Yes.

> Some people use it to extract link relations
>

Yes.

> Some people use it to generate forms
>

Yes.

> Some people use it to structure data storage.
>

But no. You MAY use JSON Schema for that, but HOW you do it is none of
JSON Schema's concern. If for a same schema you use a plain text file
to store the data, or an SQL database with joins on 4 tables, or a
Cassandra storage, or HDFS or whatever, JSON Schema doesn't give a
<beep>.

JSON Schema is not an ORM description language.

[...]

>>
>> > Suppose I want to say "Only validate one property of this instance using
>> > that schema over there" - merge/patch not only loses all semantic
>> > meaning,
>> > but it's simply incapable of solving this problem altogether.
>> >
>>
>> Good -- since JSON Schema is not meant for such things.
>
>
> Yes it is. Quoth draft-4:
>
> JSON Schema defines the media type "application/schema+json", a JSON based
> format for defining the structure of JSON data. JSON Schema provides a
> contract for what JSON data is required for a given application and how to
> interact with it. JSON Schema is intended to define validation,
> documentation, hyperlink navigation, and interaction control of JSON data.
>
>
> I'd think that pretty clearly covers the case I posted about.
>

No. See above.

[...]

>>
>> It is just meant to allow schema reuse in a MUCH more efficient manner
>> than anything existing. Without breaking backwards compatibility,
>> basing itself on standards etc.
>>
>> Also, if you use RFC 6902, you even have a "test" operation; which
>> means you can make expansion fail if some field in the source schema
>> you want to extend isn't what you expect it to be anymore!
>
>
> Your $patch/$merge is currently defined as a preprocessing step, before it
> hits the validator proper (including "test"). If you're proposing a change
> to these semantics, could you define it please?
>

It was defined in the pages I had about them, quite some time ago, but
since it didn't make it in the official list of changes these got
lost.

But that's how it is. The order of processing is, in order:

* JSON Reference,
* $merge or $patch.

It is ILLEGAL to have both $merge and $patch at the same time.