"extends"/"allOf" and schema inheritance

[Geraint (David)]

In another thread, the subject of extending schemas came up, and someone said: "There is some kind of inheritance, Jim, but not as we know it".  This is not the first time I've heard this kind of sentiment - generally, people want to specify "additionalProperties":false in one schema, but then define additional properties in another schema that extends it.

My view is that the "extends"/"allOf" behaviour is completely appropriate, and is inheritance in the purest sense.  Any other behaviour added in the name of "inheritance" would actually break the principles of inheritance instead.

I think the way inheritance works is important, and I think that the desire for different behaviour is misguided - however, I want to make sure I haven't missed something, so I want to have a discussion about it.  Let me start by laying out my case:

"B extends A"

The core of inheritance is that "B extends A" means that "Every valid B is also a valid A".  So, wherever you use an A, you should also be able to use a B.

This is actually how schema extension with "extends"/"allOf" works - if B extends A, then any data that is a valid B is also a valid A.

If the schema for A defines two keys ("key1" and "key2"), and then says "additionalProperties": false, then that is a declaration that any object that contains other keys than those two is not a valid A.  The only reason you should need to specify that is if the presence of other properties actually breaks something.

If the presence of other properties actually breaks something, then being able to define new properties in schema B makes no sense - if a piece of data actually used those properties, then it would no longer be a valid A, and it would not be able to be used in place of an A.  It would break the fundamental rule of inheritance.

"Inflating" data

The most common case I've heard of where people seem to want to specify "additionalProperties":false but still define additional properties later is when they have a set of decoders that "inflate" JSON documents into native objects.

Say you have two schemas: SchemaA and SchemaB.  You also have two classes in whatever language you're using: ClassA and ClassB.  They both have a static method inflateFromJson() which returns objects of the corresponding class.  SchemaA specifies all the constraints required such that ClassA::inflateFromJson() will not encounter an error, and the same for SchemaB/ClassB.

So - why would SchemaA ever need to specify "additionalProperties":false?  The answer is "because the ClassA::inflateFromJson() function would throw an error if there are unexpected properties".

But that's not an explanation - why was inflateFromJson() written like that?  In what other programming scenario do unknown extensions break things?  If I have a function that's expecting a ClassA object, and I give it a ClassB object, there's no way that it should throw an exception saying "I did some introspection on the object, and I don't recognise all of these methods!".

If "B extends A" in both worlds, then you absolutely should be able to give some SchemaB JSON data to ClassA::inflateFromJson() and obtain a ClassA object without any problems.

Not everyone feels the same way, though, so I wonder what I'm missing.  Thoughts?

[Heinrich Nirschl]

I fully agree with this view.

[Francis Galiegue]

On Wed, Jan 16, 2013 at 4:55 PM, Geraint (David) <gerai...@gmail.com> wrote:
> In another thread, the subject of extending schemas came up, and someone
> said: "There is some kind of inheritance, Jim, but not as we know it".

It was me ;)

I do agree with you, however this misconception about "extends" has
been the most frequently asked question on the forum, and by far. I've
had at least two issues open on my project about this.

The problem is that a lot of people understand "schema inheritance" as
"schema merging", because they intuitively refer to the fact that when
B "extends" A, B "inherits" all attributes from A. When you call
"super();" from B, it will not cause A to yell at you. But B "knows"
A.

And this is not the case with JSON Schema, since all (sub)schemas are
evaluated without having the knowledge that they have been called from
another schema (even $ref has no such knowledge, it just happens that
it can resolve somewhere within the root schema given the right URI
scope).

I really coined this phrase as a pun on a well known series, I didn't
mean to "redefine" inheritance ;)

--
Francis Galiegue, fgal...@gmail.com
"It seems obvious [...] that at least some 'business intelligence'
tools invest so much intelligence on the business side that they have
nothing left for generating SQL queries" (Stéphane Faroult, in "The
Art of SQL", ISBN 0-596-00894-5)

[Geraint (David)]

Don't worry, I got the reference - this wasn't kicked off just by that comment, I've been brewing for a while. :p

I guess what I really don't like is the idea that "schema merging" should be desirable at all, or that it addresses a valid problem.

When someone comes along with a situation like that, and says "I've tried to extend my schema, but it won't validate with the new properties!", the answer given is sometimes something like "Ah, we can't do that", as if we're missing a feature.

I'm pretty sure we're not missing a feature, though - I think the answer should definitely be "If you want to be able to extend your schema, you should not be setting "additionalProperties" to false".

[Austin Wright]

On Wednesday, January 16, 2013 8:55:45 AM UTC-7, Geraint (David) wrote:

In another thread, the subject of extending schemas came up, and someone said: "There is some kind of inheritance, Jim, but not as we know it".  This is not the first time I've heard this kind of sentiment - generally, people want to specify "additionalProperties":false in one schema, but then define additional properties in another schema that extends it.

My view is that the "extends"/"allOf" behaviour is completely appropriate, and is inheritance in the purest sense.  Any other behaviour added in the name of "inheritance" would actually break the principles of inheritance instead.

I think the way inheritance works is important, and I think that the desire for different behaviour is misguided - however, I want to make sure I haven't missed something, so I want to have a discussion about it.  Let me start by laying out my case:

"B extends A"

The core of inheritance is that "B extends A" means that "Every valid B is also a valid A".  So, wherever you use an A, you should also be able to use a B.

This was a bit confusing to comprehend without describing as a set. If B extends A, you're saying that B is a subset of A, therefore every B is also an A.

I'm thinking there's some confusion because it is also desirable to use "extends" to create supersets, that is, use a JSON Schema to implement re-usable components of a schema, where every A is also a valid B, because B extends A as a superset. You would do this by saying, "B is A with these additional defined properties." You're overriding old schemas, "Instead of using A's schema for the 'bProp' property (which is must not exist, because it is not in properties or patternProperties and additionalItems is false), use this schema instead."

 

This is actually how schema extension with "extends"/"allOf" works - if B extends A, then any data that is a valid B is also a valid A.

If the schema for A defines two keys ("key1" and "key2"), and then says "additionalProperties": false, then that is a declaration that any object that contains other keys than those two is not a valid A.  The only reason you should need to specify that is if the presence of other properties actually breaks something.

If the presence of other properties actually breaks something, then being able to define new properties in schema B makes no sense - if a piece of data actually used those properties, then it would no longer be a valid A, and it would not be able to be used in place of an A.  It would break the fundamental rule of inheritance.

"Inflating" data

The most common case I've heard of where people seem to want to specify "additionalProperties":false but still define additional properties later is when they have a set of decoders that "inflate" JSON documents into native objects.

Say you have two schemas: SchemaA and SchemaB.  You also have two classes in whatever language you're using: ClassA and ClassB.  They both have a static method inflateFromJson() which returns objects of the corresponding class.  SchemaA specifies all the constraints required such that ClassA::inflateFromJson() will not encounter an error, and the same for SchemaB/ClassB.

So - why would SchemaA ever need to specify "additionalProperties":false?  The answer is "because the ClassA::inflateFromJson() function would throw an error if there are unexpected properties".

But that's not an explanation - why was inflateFromJson() written like that?  In what other programming scenario do unknown extensions break things?  If I have a function that's expecting a ClassA object, and I give it a ClassB object, there's no way that it should throw an exception saying "I did some introspection on the object, and I don't recognise all of these methods!".

If "B extends A" in both worlds, then you absolutely should be able to give some SchemaB JSON data to ClassA::inflateFromJson() and obtain a ClassA object without any problems.

Not everyone feels the same way, though, so I wonder what I'm missing.  Thoughts?

You would use additionalProperties = false when, well, you have no way of handling additional properties. For instance, if I'm accepting JSON data and storing it in a database, my JSON schema contains metadata about which column and table to store the data in. By specifying additionalProperties: false, I'm declaring that I have no way of handling the incoming data with unknown properties.

Let's say from time to time I publish new versions of my schema as my API and tables update. Like any good HTTP API, I store this revision information as a schema (Content-Type), not in my resource URLs (which would imply a version on the resource, not on the API -- this should be obvious, but even Twitter does this, so idk). I also have a URL which by definition refers to the latest API I have published. Normally you want to use the URL for a particular schema version, but here, someone wants to publish a schema saying "this JSON schema is the same as one of these schema, even if a new one is published at a later date, except it re-defines properties within the schema, removing 'must not be defined' (by virtue of formerly having no listing in 'properties' or 'patternProperties'), and adds the restriction 'must be an integer'." This is why you would want to extend or redefine an existing schema, instead of making a copy and deleting the parts you want to change.

However, if there is no way to extend a schema in a way that creates supersets (by deleting definitions as well as adding new restrictions -- and defining new properties, when the old schema has additionalProperties:false, effectively deletes old properties), then specifying this sort of behavior is impossible.

This isn't just a problem for additionalProperties:false. This is a problem whenever additionalProperties adds any restrictions at all (e.g. patternProperties:{type:'string'} and later extending a property which must be an integer, or 'all'). Iirc patternProperties applies to a property even when explicitly named in 'properties', but if it does not, then this applies to patternProperties too.

There's useful cases for both. It's useful to be able to create a schema that creates a superset of valid values from another schema, but it's also useful to know, for computational purposes, that 'extends' can only create subsets. We should look more closely into the semantics and introduce new properties if necessary, types of extends properties that are defined to only create subsets (which is 'allOf'), and properties which may re-define properties based off of another schema (which should be 'extends').

[Geraint (David)]

Ah, a mathematician!  :)

Yes - the rules of inheritance define a subset of the set of possible values.  If have a function that takes a ClassA as an argument, then the set of possible arguments is every possible object of every class that is or extends ClassA.  If ClassB extends ClassA, then the set of possible ClassB objects is indeed a subset of that.

On Thursday, January 17, 2013 11:29:57 PM UTC, Austin Wright wrote:

You would use additionalProperties = false when, well, you have no way of handling additional properties. For instance, if I'm accepting JSON data and storing it in a database, my JSON schema contains metadata about which column and table to store the data in. By specifying additionalProperties: false, I'm declaring that I have no way of handling the incoming data with unknown properties.

OK - let's look at the classical/OO equivalent of this situation.  You have a function storeAInDb(ClassA a).  Say, that this function calls getter methods on ClassA objects (title(), message(), email()) and stores the results in the database.

Now, if I pass in a ClassB to this function, there will be properties of the incoming data that the function has no way of handling.  So is there a way of making sure that our method only takes in ClassA, and not any subclasses that define additional methods?  What is the OO equivalent of specifying "additionalProperties":false in this case?

I would argue that the equivalent is declaring ClassA to be "final" - disallowing any subclasses, because they might contain more information than the original.  Fine - there are sometimes reasons to do that.  But this desire for schema merging is the equivalent of declaring your class "final" because it exactly represents a row in a database, and then saying "Oh, no!  It won't let me extend my final class!  The language must be missing a feature!".

On Thursday, January 17, 2013 11:29:57 PM UTC, Austin Wright wrote:

However, if there is no way to extend a schema in a way that creates supersets (by deleting definitions as well as adding new restrictions -- and defining new properties, when the old schema has additionalProperties:false, effectively deletes old properties), then specifying this sort of behavior is impossible.

Let's look at what the classical OO equivalent of creating-supersets is.  Sub-sets are inheritance, so creating a super-set is the equivalent of creating a new super-class for an existing class.

So the situation you have is kind of like this:  A is a base class, and B and C extend A.  C is final (marked with a "*") because it represents an exact set of data (e.g. a database row) and functions that deal with it cannot handle additional properties/methods).

   A  (superclass)

  / \

 B   C*

But you then want to re-use the things you defined in C, relaxing some restrictions.  So you create a "superset" (super-class), D:

   A   D

  / \ /

 B   C*

Now, there are languages that let you do similar things in OO (Objective C, for instance).  But it is not an essential feature - many languages do not support multiple-inheritance, let alone inserting new super-classes after the fact.

Most people who have wanted the feature have been attempting to create something akin to classical inheritance.  But this is absolutely not classical inheritance, and I don't think we should even consider the feature unless the people asking for it actually know what it means.

On Thursday, January 17, 2013 11:29:57 PM UTC, Austin Wright wrote:

You're overriding old schemas, "Instead of using A's schema for the 'bProp' property (which is must not exist, because it is not in properties or patternProperties and additionalItems is false), use this schema instead."

 

OK - but this is not inheritance in any way.  It is code re-use, but not inheritance.

On Thursday, January 17, 2013 11:29:57 PM UTC, Austin Wright wrote:

We should look more closely into the semantics and introduce new properties if necessary, types of extends properties that are defined to only create subsets (which is 'allOf'), and properties which may re-define properties based off of another schema (which should be 'extends').

I completely disagree - as I've said above, generally people who want this feature are looking for inheritance.  But this is not inheritance, so calling it "extends" will just add more fuel to the fire.

I would be more receptive if people were proposing a keyword called "reusePropertiesFrom" or "inspiredBy" - but I still don't think it's a feature we need, any more than I need to extend final classes in Java.

[Austin Wright]

No matter what, 'allOf' implements your 'extends', the ability to create a schema validating only a subset of values: That in addition to the properties defined by this other, extended schema, it must also validate against the locally defined schemas.

So it seems very natural that any other similar term would allow you to do the opposite and re-define properties. I don't think 'extends' necessarily implies classical inheritance, or that such usage is confusing anyone, but perhaps there is a better term, maybe 'implements'. It's worth giving some effort towards, but I wouldn't be too concerned about choosing the right word for the job, as long as it's well-defined and internally consistent. For instance, I haven't seen anyone get confused that, even though an ECMAScript Array is also an Object, a JSON array is never an object.

I mean, is there any evidence to suggest that the v3 draft definition confused anyone more than some alternative term?

If multiple inheritance isn't often used, it's because often you plain don't need it. But when it is used I don't see anyone getting confused that its use of 'extends' creates a union, not an intersection, of the inherited classes.

All of the other similar terms I can think of have stronger connotations - 'parent' (implies a relationship), 'inherits' (where something is inherited there must be some parent), 'derivedFrom' (maybe), 'reuse' (not really applicable to schemas) , 'base' (URIs). So I think I'll argue for using 'extends'. And typically, you use an 'extends' keyword in a programming language to define new functionality, new methods or properties. Obviously, JSON doesn't have methods, the only thing 'adding new functionality' could mean is that you expand the range of legal values, or at least better define them in some capacity (which may even mean a mix of allowing new instance values and restricting others).

And perhaps I've been programming PHP too long but in C++ I believe you may define new properties in a subclass that aren't accessible from the superclass. Is the legal range of values for an instance of the subclass not a superset of the legal range of values for an instance of the superclass? (The fact that you can't use an instance of the superclass where an instance of the subclass is called for is immaterial.)

Austin Wright.

[Francis Galiegue]

On Fri, Jan 18, 2013 at 7:02 PM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...]

>
> No matter what, 'allOf' implements your 'extends', the ability to create a
> schema validating only a subset of values: That in addition to the
> properties defined by this other, extended schema, it must also validate
> against the locally defined schemas.
>
> So it seems very natural that any other similar term would allow you to do
> the opposite and re-define properties. I don't think 'extends' necessarily
> implies classical inheritance, or that such usage is confusing anyone, but
> perhaps there is a better term, maybe 'implements'. It's worth giving some
> effort towards, but I wouldn't be too concerned about choosing the right
> word for the job, as long as it's well-defined and internally consistent.
> For instance, I haven't seen anyone get confused that, even though an
> ECMAScript Array is also an Object, a JSON array is never an object.
>
> I mean, is there any evidence to suggest that the v3 draft definition
> confused anyone more than some alternative term?
>

Well, it certainly got people confused, nobody can argue against that.
On my project alone, I remember offhand of two opened issues which
were linked to "extends". I've had to explain each time that this was
not a bug ;)

This problem is so frequent that I tried and wrote some (unfinished)
rules for what I coined as "schema merging":

https://github.com/json-schema/json-schema/wiki/Schema-merging-rules

I see this as a good idea, personally, although some will not agree
with me. But it certainly addresses this problem.

[Austin Wright]

On Friday, January 18, 2013 11:12:01 AM UTC-7, fge wrote:

Well, it certainly got people confused, nobody can argue against that.
On my project alone, I remember offhand of two opened issues which
were linked to "extends". I've had to explain each time that this was
not a bug ;)

Well now I'm just confused. What's the purpose of 'extends' if it's exactly the same as 'type'? Or rather, how are they different? I guess 'extends' appears to have been renamed to 'allOf', and 'type' to 'anyOf', which is a very suitable decision, considering most people seemed to expect 'extends' to work as I described above (and as I believed it to work):

https://github.com/fge/json-schema-validator/issues/21 This person reported exactly what I argued an 'extends' property should do, too.

The reason I thought it worked like that is the maintainer of the library, while I was contributing to it, wrote 'extends' property support that did exactly this, replacing old schemas with new schemas (using recursive assignment), instead of checking for conformance to both old and new schemas. And it passed all the tests in the then-not-as-official suite.

 

This problem is so frequent that I tried and wrote some (unfinished)
rules for what I coined as "schema merging":

https://github.com/json-schema/json-schema/wiki/Schema-merging-rules

I see this as a good idea, personally, although some will not agree
with me. But it certainly addresses this problem.

If your theoretical algorithm is commutative, then how can it possibly be useful? It just seems to define a way to re-write a schema that uses 'allOf' or 'anyOf' into an equivalent schema that lacks those properties. Inheritance is by definition is not commutative, 'Take A and extend with B' is not necessarily the same as 'Take B and extend with A'.

[Francis Galiegue]

On Fri, Jan 18, 2013 at 7:53 PM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
>
>
> On Friday, January 18, 2013 11:12:01 AM UTC-7, fge wrote:
>>
>> Well, it certainly got people confused, nobody can argue against that.
>> On my project alone, I remember offhand of two opened issues which
>> were linked to "extends". I've had to explain each time that this was
>> not a bug ;)
>
>
> Well now I'm just confused. What's the purpose of 'extends' if it's exactly
> the same as 'type'? Or rather, how are they different? I guess 'extends'
> appears to have been renamed to 'allOf', and 'type' to 'anyOf', which is a
> very suitable decision, considering most people seemed to expect 'extends'
> to work as I described above (and as I believed it to work):
>
> https://github.com/fge/json-schema-validator/issues/21 This person reported
> exactly what I argued an 'extends' property should do, too.

If we are talking draft v3:

{ "type": [ { "schema1": "here" }, { "schema2": "here" } ] }

means that the instance must obey either of schema1, or schema2, or
both. With schema1 and schema2 evaluated _independently_.

When using:

{
"schema1": "here",
"extends": { "schema2": "here" }
}

the meaning completely changes: it means all constraints of schema2
must be obeyed _in addition to_ all constraints of schema1. And again,
schema1 and schema2 are evaluated _INDEPENDENTLY OF ONE ANOTHER_.

So, if your schema2 defines a schema for property p and has set
additionalProperties to false, then your q property defined in schema1
will not change a single thing.

>
> The reason I thought it worked like that is the maintainer of the library,
> while I was contributing to it, wrote 'extends' property support that did
> exactly this, replacing old schemas with new schemas (using recursive
> assignment), instead of checking for conformance to both old and new
> schemas. And it passed all the tests in the then-not-as-official suite.
>

Draft v3 agrees with my (and David's) definition. It says this about extends:

----
The inheritance rules are such that any instance that is valid
according to the current schema MUST be valid according to the
referenced schema. [or schemas if the value is an array of schemas]
----

So, an instance valid against schema1 in the example above must also
be valid against schema2. And schema2 has no notion at all that it was
called from schema1.

[...]

>
> If your theoretical algorithm is commutative, then how can it possibly be
> useful? It just seems to define a way to re-write a schema that uses 'allOf'
> or 'anyOf' into an equivalent schema that lacks those properties.

Wrong, see above.

[Austin Wright]

Oh, I understand what it's doing now (I think). It's specifying a list of schemas which it must also be valid against, it's the equivalent of allOf, yes?

My argument is that people expect 'extends' to work in the way that I'm arguing, i.e. the property that allows one to override portions of old schemas should be called 'extends', and I supplied evidence that using 'allOf'/the strict subset behavior is what is confusing for people.

 

Wrong, see above.

What above? I mean, I'm not sure what goal your theoretical algorithm solves. If I want to use an old schema, except change 'maxItems' to a smaller or larger value, this algorithm, because it is commutative, only allows me to go in one direction. I'll be able to make maxItems bigger but not smaller, or vice-versa. This isn't very useful.

[Heinrich Nirschl]

On Saturday, January 19, 2013 9:54:11 AM UTC+1, Austin Wright wrote:

I mean, I'm not sure what goal your theoretical algorithm solves. If I want to use an old schema, except change 'maxItems' to a smaller or larger value, this algorithm, because it is commutative, only allows me to go in one direction. I'll be able to make maxItems bigger but not smaller, or vice-versa. This isn't very useful.

I think it is useful. If you specify B extends A you want to make sure that a program expecting an A will still work when it gets a B. So there is an IsA relation between B and A. If you were allowed to increase the maxItems specified in A you could get a buffer overflow.

What you want to have is also useful, its kind of I want to base may definition on schema A but with some changes. However this not inheritance, it is "reuse with patches".

[Geraint (David)]

I think "extends" definitely implies classical inheritance.

 

, but perhaps there is a better term, maybe 'implements'. It's worth giving some effort towards, but I wouldn't be too concerned about choosing the right word for the job, as long as it's well-defined and internally consistent. For instance, I haven't seen anyone get confused that, even though an ECMAScript Array is also an Object, a JSON array is never an object.

Actually - yes, that most definitely happened.  I can't remember where, but someone came along with a schema that said {"type": "object", "items": ...} and was surprised it wasn't validating things correctly.

I looked at the blog post they'd been taking as an example, and it said to do exactly that, because "Remember in JavaScript, all arrays are objects!".  It emailed him and he changed it, but it is most definitely a mistake people have made.

 

I mean, is there any evidence to suggest that the v3 draft definition confused anyone more than some alternative term?

If multiple inheritance isn't often used, it's because often you plain don't need it. But when it is used I don't see anyone getting confused that its use of 'extends' creates a union, not an intersection, of the inherited classes.

I you think in terms of sets of possible values/objects (as we have been), then it is an intersection.  But in classical OO with multiple inheritance, if you look at the set of defined methods/properties, it's a union.

We've both approached this from a mathematical perspective, but not everyone thinks in terms of set theory.

 

All of the other similar terms I can think of have stronger connotations - 'parent' (implies a relationship), 'inherits' (where something is inherited there must be some parent), 'derivedFrom' (maybe), 'reuse' (not really applicable to schemas) , 'base' (URIs). So I think I'll argue for using 'extends'. And typically, you use an 'extends' keyword in a programming language to define new functionality, new methods or properties. Obviously, JSON doesn't have methods, the only thing 'adding new functionality' could mean is that you expand the range of legal values, or at least better define them in some capacity (which may even mean a mix of allowing new instance values and restricting others).

This is because "extends" in a programming language refers to the system of classical inheritance.  In classical OO, this refers to the set of methods that are guaranteed to be available (by restricting the value to the subset of objects that implement that method).

But crucial to this metaphor is that you haven't defined your class to be "final" (additionalProperties: false).  In set theoretic terms, they are equivalent - you are placing constraints on the properties/methods you haven't defined yet - specifically, by saying they cannot exist.

 

And perhaps I've been programming PHP too long but in C++ I believe you may define new properties in a subclass that aren't accessible from the superclass. Is the legal range of values for an instance of the subclass not a superset of the legal range of values for an instance of the superclass? (The fact that you can't use an instance of the superclass where an instance of the subclass is called for is immaterial.)

Um.  No?  Every legal value for an instance of the subclass is a legal value for an instance of the superclass, but the inverse is not true.  So the legal range of values for a subclass is a subset of the legal range of values for the superclass.

In terms of the set of possible values, classical inheritance defines a subset.

 

Austin Wright.

[Geraint (David)]

"reuse with patches" is dead on.  Nicely put!

In fact, I think "patches" would be exactly the right keyword to describe this behaviour.

Geraint

[Francis Galiegue]

On Sat, Jan 19, 2013 at 9:54 AM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...]
>>

>> So, an instance valid against schema1 in the example above must also
>> be valid against schema2. And schema2 has no notion at all that it was
>> called from schema1.
>
>
> Oh, I understand what it's doing now (I think). It's specifying a list of
> schemas which it must also be valid against, it's the equivalent of allOf,
> yes?
>

Yes. And...

> My argument is that people expect 'extends' to work in the way that I'm
> arguing

Yes :/

> i.e. the property that allows one to override portions of old
> schemas should be called 'extends', and I supplied evidence that using
> 'allOf'/the strict subset behavior is what is confusing for people.
>

Well, allOf has been introduced to kill this confusion precisely.
Along with anyOf to replace type: [ {}, {} ] which was quite ungainly,
and oneOf, which has no equivalent in v3.

[...}

>
> What above? I mean, I'm not sure what goal your theoretical algorithm
> solves. If I want to use an old schema, except change 'maxItems' to a
> smaller or larger value, this algorithm, because it is commutative, only
> allows me to go in one direction. I'll be able to make maxItems bigger but
> not smaller, or vice-versa. This isn't very useful.
>

Hey, we cannot have everything :p

That is still uncharted territory, but for instance, in your use case,
it does what you expect it does. Merging:

{
"properties": { "p": {} },
"additionalProperties": false
}

and:

{
"properties": { "q": {} }
}

gives:

{
"properties": { "p": {}, "q": {} },
"additionalProperties": false
}

This is what people actually expected "extends" to do. And it should
work both ways, yes. Anyway, it needs work. David doesn't like it,
personally I think it is dead useful.

[Francis Galiegue]

On Sat, Jan 19, 2013 at 11:38 AM, Geraint (David) <gerai...@gmail.com> wrote:
> "reuse with patches" is dead on. Nicely put!
>
> In fact, I think "patches" would be exactly the right keyword to describe
> this behaviour.
>

Ick... There is also JSON Patch coming along, I believe using this as
a keyword could cause potential confusion.

[Francis Galiegue]

On Sat, Jan 19, 2013 at 11:59 AM, Francis Galiegue <fgal...@gmail.com> wrote:
[...]

>
> This is what people actually expected "extends" to do. And it should
> work both ways, yes. Anyway, it needs work. David doesn't like it,
> personally I think it is dead useful.
>

On second thoughts, I think I am going the wrong way. The approach
coined by Austin, David and Heinrich sounds better. One problem I had
will full merging like I thought is that some keywords were
unmergeable, or very difficult to do (how to merge "items" if schema 1
has an array and schema2 has a single schema?).

I think I could even have a go at it later on.

[Austin Wright]

On Saturday, January 19, 2013 3:33:37 AM UTC-7, Geraint (David) wrote:

Um.  No?  Every legal value for an instance of the subclass is a legal value for an instance of the superclass, but the inverse is not true.  So the legal range of values for a subclass is a subset of the legal range of values for the superclass.

In terms of the set of possible values, classical inheritance defines a subset.

I had to try it myself: It appears, at least in the C++ inheritance model, inheritance indeed allows subclasses to extend their range of values:

#include <stdio.h>

class Super {

public:

        int width;

        int height;

        void log2(){

                printf("(%d, %d)\n", width, height);

        }

};

class Sub: public Super {

public:

        int depth;

        void log3 (){

                printf("(%d, %d, %d)\n", width, height, depth);

        }

};

int main () {

        Sub a;

        a.width = 0;

        a.height = 2;

        a.depth = 5;

        a.log2();

        a.log3();

        return 0;

}

$ ./a.out

(0, 2)

(0, 2, 5)

The legal range of values for an instance of Sub is definitely a superset of the values allowed in an instance of Super.

JSON deals only in instances, therefore we would expect an extended schema to allow a superset of values in an instance (though adding restrictions is acceptable, too). This is the behavior that people expect when they see the word "extends", the ability to define new properties in their sub-schema that would not otherwise be permitted in the super-schema.

[Austin Wright]

On Saturday, January 19, 2013 4:10:16 AM UTC-7, fge wrote:

On Sat, Jan 19, 2013 at 11:38 AM, Geraint (David) <gerai...@gmail.com> wrote:
> "reuse with patches" is dead on.  Nicely put!
>
> In fact, I think "patches" would be exactly the right keyword to describe
> this behaviour.
>

Ick... There is also JSON Patch coming along, I believe using this as
a keyword could cause potential confusion.

 

Yeah, while it's related behavior, I definitely wouldn't use the term "patch", or rather "patchFrom", or anything similar. I'd expect the behavior of a "patch" property to be much more advanced than what anyone has discussed here.

[Francis Galiegue]

On Sat, Jan 19, 2013 at 4:40 PM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...]

>This is the behavior that people expect when they see the
> word "extends", the ability to define new properties in their sub-schema
> that would not otherwise be permitted in the super-schema.
>

Yes, people expect that. No, this is not how extends works, and draft
v3 wording tells that although you really need to read carefully.

Anyway. I don't think discussing the theory at hours on end will be of
benefit at this point. It is clearly the case that people who didn't
pay close attention to what the draft said have misunderstood extends,
and the next step from here on is to devise a solution which is
technically sound.

[Francis Galiegue]

On Sat, Jan 19, 2013 at 4:45 PM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...]

>>
>> Ick... There is also JSON Patch coming along, I believe using this as
>> a keyword could cause potential confusion.
>
> Yeah, while it's related behavior, I definitely wouldn't use the term
> "patch", or rather "patchFrom", or anything similar. I'd expect the behavior
> of a "patch" property to be much more advanced than what anyone has
> discussed here.
>

<biased by="Java">I was thinking of "overrides".</biased> But that is
for February, right now drafts need finalizing :p

[Austin Wright]

On Saturday, January 19, 2013 8:46:18 AM UTC-7, fge wrote:

Yes, people expect that. No, this is not how extends works, and draft
v3 wording tells that although you really need to read carefully.

Anyway. I don't think discussing the theory at hours on end will be of
benefit at this point. It is clearly the case that people who didn't
pay close attention to what the draft said have misunderstood extends,
and the next step from here on is to devise a solution which is
technically sound.

Well, right, this is discussing what functionality an extensible schema should have, not how a particular draft works in the positive.

[Heinrich Nirschl]

Actually, it is a subset. Take a pointer of the superclass

Super *p;

You can not only assign pointers to instances of Super, but also pointers to instances of Sub, and of any other class that inherits from Super, say Sub2.

On the other hand this pointer

Sub *q;

does not accept pointers to Super or Sub2 instances (unless Sub2 is also a subclass of Sub) so it restricts the values to a subset.

[Francis Galiegue]

On Sat, Jan 19, 2013 at 4:51 PM, Austin Wright
<diamon...@users.sourceforge.net> wrote:
[...]
>
>

> Well, right, this is discussing what functionality an extensible schema
> should have, not how a particular draft works in the positive.
>

OK, can we just please stop using vocabulary related to "extends"?
This keyword has polluted, and continues to pollute, people's minds,
especially those who are seasoned with OO programming. It is to the
point that I feel twitchy each time I read these words in the context
of JSON Schema, given the time I, and others, have spent "dispelling
the myth".

We need to find another vocabulary. "Inheritance", "overriding",
"reuse", whatever. But please, nothing "extends"-related :p

[Austin Wright]

On Saturday, January 19, 2013 8:57:15 AM UTC-7, Heinrich Nirschl wrote:

Actually, it is a subset. Take a pointer of the superclass

Super *p;

You can not only assign pointers to instances of Super, but also pointers to instances of Sub, and of any other class that inherits from Super, say Sub2.

On the other hand this pointer

Sub *q;

does not accept pointers to Super or Sub2 instances (unless Sub2 is also a subclass of Sub) so it restricts the values to a subset.

 

I'm not talking about membership, a member of a subclass is by definition a member of the superclass. However JSON has no such concept as membership or even inheritance, so this is immaterial. I'm talking about the value range of an instance, which is what JSON Schema deals in. And like in inheritance, where the value range of instances of subclasses are (by default) a superset of the value range of instances of superclasses, so too should an extends property validate a superset of values.

[Austin Wright]

On Saturday, January 19, 2013 8:57:48 AM UTC-7, fge wrote:

OK, can we just please stop using vocabulary related to "extends"?
This keyword has polluted, and continues to pollute, people's minds,
especially those who are seasoned with OO programming. It is to the
point that I feel twitchy each time I read these words in the context
of JSON Schema, given the time I, and others, have spent "dispelling
the myth".

We need to find another vocabulary. "Inheritance", "overriding",
"reuse", whatever. But please, nothing "extends"-related :p

Idk, I thought I made a pretty good case that the behavior I am arguing should exist is "extends": That it works like "extends" keywords and inheritance in classical OO programming (unlike the v3 definition), and that the only bugs relating to "extends" in your library are people who expected it to function with this behavior.

"Inheritance" is even worse because JSON doesn't have inheritance, there's no such thing as being a "member of a schema" ("instance of a schema" just means "validates against a schema", two separate but identical schemas share the same instances, unlike OO).

"Overriding" would be a better term for the behavior being described, let's adopt that. But "extends" still sounds like the superior property name... unless you wanted "overridesFrom" which isn't as clear what that does.

[Francis Galiegue]

On Sun, Jan 20, 2013 at 7:03 AM, Austin Wright <diamon...@users.sourceforge.net> wrote:
[...]
>

> Idk, I thought I made a pretty good case that the behavior I am arguing
> should exist is "extends"
>

OK, I'll say it one last time: draft v3 disagrees with you. Quoting the appropriate section again (emphasis mine -- it is the first time I have sent an HTML mail in my whole life):

----
5.26.  extends

   The value of this property MUST be another schema which will provide
   a base schema which the current schema will inherit from.  The

   inheritance rules are such that any instance that is valid according
   to the current schema MUST be valid according to the referenced

   schema.  This MAY also be an array, in which case, the instance MUST
   be valid for all the schemas in the array.  A schema that extends
   another schema MAY define additional attributes, constrain existing
   attributes, or add other constraints.

   Conceptually, the behavior of extends can be seen as validating an
   instance against all constraints in the extending schema as well as
   the extended schema(s).
---

That is clear enough, it seems.

[Francis Galiegue]

Sorry, I completely misread your mail :(

Please forgive me.

[Heinrich Nirschl]

It seems to me that the difficulty comes from the different nature of the specification of allowed values. In languages like C++ if you add something to your class definition than you add also a field to your instance. However, JSON schema is subtractive, if you add a keyword to the schema you restrict your value set further and make it smaller (or it stays the same).

Adding an additional property to a schema can have both effects, if "additionalProperties" is false, it will reduce the value set, because you add a restriction on the property value that was not there before. However if "additionalProperties" is true, it will increase the value set, because now also values with this new property are allowed.

So it is a good move to eliminate the "extends". One would never intuitively understand what is extended, the value set or the schema text.

[Heinrich Nirschl]

Argh, please reverse true and false in the lust but one paragraph.

[Austin Wright]

On Sunday, January 20, 2013 12:56:17 AM UTC-7, Heinrich Nirschl wrote:

Adding an additional property to a schema can have both effects, if "additionalProperties" is false, it will reduce the value set, because you add a restriction on the property value that was not there before. However if "additionalProperties" is true, it will increase the value set, because now also values with this new property are allowed.

You can do either in OO models, sometimes, too. If you want to define that a property defined in the superclass has a smaller range in the subclass, often you can do that.

I agree there's going to be some confusion if this overrides behavior were called with an "extends" property, if only because that term is defined differently in v3. But that's no reason to not fix the definition with something more in-line with what people are expecting. So long as there is no better term.

Well, maybe not. Maybe "supplement"? idk, that's not strong enough a word.

[Francis Galiegue]

OK, so, first of all we need to find an appropriate keyword. What has
been seen so far, and some random others which popped out of my mind:

* "supplements",
* "overrides",
* "mergeWith",
* "reuse" (or "reusing"?),
* "basedOn".

Others?

Oh, and of course, once we have the keyword, we have to define it
appropriately. Fun :p

[Austin Wright]

Perhaps we should spec out the behavior for overrides functionality in a new thread, then we can name it. In most cases the functionality is obvious, but what to do about arrays, etc, what if you want to re-specify the entire array or instead add new elements, or re-define just one element.

Also: "prototype"!

[Geraint (David)]

On Sunday, January 20, 2013 9:49:03 AM UTC, Austin Wright wrote:

On Sunday, January 20, 2013 12:56:17 AM UTC-7, Heinrich Nirschl wrote:

Adding an additional property to a schema can have both effects, if "additionalProperties" is false, it will reduce the value set, because you add a restriction on the property value that was not there before. However if "additionalProperties" is true, it will increase the value set, because now also values with this new property are allowed.

You can do either in OO models, sometimes, too. If you want to define that a property defined in the superclass has a smaller range in the subclass, often you can do that.

Yeah - that's creating a subset of the possible values.  But with your proposed idea of "schema overriding" or whatever, you can do the equivalent of defining a property in the subclass that has a wider range of values than the super-class - and that is not possible in OO models.

I'm still not clear why you want this feature.  "additionalProperties":false is like a final class in Java - instead of trying to invent a way to nullify the "final" keyword, you should really be lookiing at your model, because something is obviously wrong there.
 

[Geraint (David)]

I also think that even if it were useful (which I don't see how it is), it is going to be conceptually (and possibly syntactically) complicated.  I think that with enough debate, you could probably work out rules and a syntax, but there's the danger that it'll be difficult to understand and inelegant to implement.

Recently, I wrote a (pretty readable) validator for v4 in 2 hours and less than 500 lines.  No support for "$ref", but still - the current set of validation keywords are very simple to implement.

This was largely because the current schema keywords are all "context-free".  That is an incredibly good property for a validation algorithm to have - you can just recursively iterate over the existing data/schemas without tracking how you got there at all (or creating new pseudo-schemas, or whatever).

But by definition, this "schema overriding" is not context-free, and so I have all the same objections to it as I do for the idea of Schema Templating (which is also a proposed context-sensitive extension to the standard aimed at code re-use).

Now - I don't have an objection to people working out how such a feature might work hypothetically (same with Schema Templating).  People getting excited about the standard and the ways to extend it is good!  But I am not yet convinced that this feature is useful, or that it won't compromise some very useful properties of the existing standard.

[Geraint (David)]

Austin - can I just ask what definition you are using for "value range"?  Because I don't see how a value range for a subclass can ever be a superset of a value range for the superclass.

The JSON Schema model of inheritance is similar to the OO idea of "duck typing" - you judge by the values/properties/methods present, not the actual definitions.

But even with that model, the "value range" for a subclass (i.e. the set of possible values that can convincingly pass as instances of the subclass) is a subset of the "value range" for the superclass.

[Chris Miles]

When you need A to have additionalProperties: false but also want to extend it, then introduce B that matches A except for additionalProperties:true and make A extend B with additionalProperties: false.

Make C extend B with new properties as required. B allows additional properties so an instance of C can match schema B and schema C.

So A is constrained and C is like A but with new properties.

I have found this pattern very useful. Initially it seems unnatural but you can get used to it :)

Chris

On 21/01/13 14:06, Geraint (David) wrote:

I'm still not clear why you want this feature.  "additionalProperties":false is like a final class in Java - instead of trying to invent a way to nullify the "final" keyword, you should really be lookiing at your model, because something is obviously wrong there.

 

I agree there's going to be some confusion if this overrides behavior were called with an "extends" property, if only because that term is defined differently in v3. But that's no reason to not fix the definition with something more in-line with what people are expecting. So long as there is no better term.

Well, maybe not. Maybe "supplement"? idk, that's not strong enough a word.

--
 
 

[Geraint (David)]

Sounds like a good solution to the problem to me.  You have the schema that defines the properties, and separately, you have a schema that restricts the properties.  You can then extend one without touching the other.

Worth noting, though, that you will need to list all the property names in A as well, otherwise no property names should be allowed.  "additionalProperties" applies to all the properties not otherwise covered in the immediate schema, not other included schemas.

[Austin Wright]

On Monday, January 21, 2013 7:06:11 AM UTC-7, Geraint (David) wrote:

Yeah - that's creating a subset of the possible values.  But with your proposed idea of "schema overriding" or whatever, you can do the equivalent of defining a property in the subclass that has a wider range of values than the super-class - and that is not possible in OO models.

I posted a 20-line C++ file that proves the opposite. You may define properties in subclasses that are not permitted in superclasses. Superclasses cannot be used where an instance of a subclass is called for precisely because of this reason: The superclass will not have the required properties defined.

So: The set of possible values for a subclass instance is a superset of the possible range of values for a superclass.

[Austin Wright]

On Monday, January 21, 2013 7:06:11 AM UTC-7, Geraint (David) wrote:

On Sunday, January 20, 2013 9:49:03 AM UTC, Austin Wright wrote:

On Sunday, January 20, 2013 12:56:17 AM UTC-7, Heinrich Nirschl wrote:

Adding an additional property to a schema can have both effects, if "additionalProperties" is false, it will reduce the value set, because you add a restriction on the property value that was not there before. However if "additionalProperties" is true, it will increase the value set, because now also values with this new property are allowed.

You can do either in OO models, sometimes, too. If you want to define that a property defined in the superclass has a smaller range in the subclass, often you can do that.

Yeah - that's creating a subset of the possible values.  But with your proposed idea of "schema overriding" or whatever, you can do the equivalent of defining a property in the subclass that has a wider range of values than the super-class - and that is not possible in OO models.

I'm still not clear why you want this feature.  "additionalProperties":false is like a final class in Java - instead of trying to invent a way to nullify the "final" keyword, you should really be lookiing at your model, because something is obviously wrong there.

Also, the behavior I and most people are interested in is defining new, previously undefined properties. But your remark about re-defining existing properties is of course correct. Maybe we don't permit re-defining existing properties. (Assuming this belongs in the spec at all, as you point out.)

[Geraint (David)]

You're quite obviously using a different meaning for "range of values" from me, which is why I asked the question.

Are you talking about the "range of properties that can be guaranteed to exist"?  Or "range of values that can be considered valid instances of class X"?

If it's the second one, then I don't see how you're seeing a superset there - any valid instances of the subclass are valid instances of the superclass as well.

If it's the first one, then what you're talking about is not analogous to "which properties are allowed to be defined in my data".  It's analogous to "which properties are guaranteed to exist in my data/class".  The equivalent to that in JSON Schema terms is being able to add more properties to the set of required properties (using the "required" keyword) - which you can already do with the current spec.

[Austin Wright]

On Monday, January 21, 2013 7:29:03 AM UTC-7, Geraint (David) wrote:

Austin - can I just ask what definition you are using for "value range"?  Because I don't see how a value range for a subclass can ever be a superset of a value range for the superclass.

The JSON Schema model of inheritance is similar to the OO idea of "duck typing" - you judge by the values/properties/methods present, not the actual definitions.

But even with that model, the "value range" for a subclass (i.e. the set of possible values that can convincingly pass as instances of the subclass) is a subset of the "value range" for the superclass.

Take my C++ Example, and let int be the set of values for a variable of type int (whatever it happens to be exactly).

The value range of an instance of Super is (int x int). Sub has an additional dimension defined in addition to the properties defined by its parent Super, making its value range (int x int x int). When accessed as a member of Super, it is merely (int x int), or perhaps (int x int x undefined), therefore I would call the range of values, the set of possible values for an instance of Sub, a superset of the set of possible values for an instance of Super.

Or to put it another way, there exists a serialized data structure that is a valid Sub, but not a valid Super, but no such structure the other way around. Every instance of Super may be read into an instance of Sub, but an instance of Sub can not be unserialized and read into a Super instance.

[Geraint (David)]

OK - I think I might have got the confusion figured out.

In the OO world, what you are interested in is the properties that are guaranteed to exist.  Sub-classes might define new methods or whatever, but they cannot make methods/properties protected or private if they were public in a super-class.

The equivalent in JSON Schema is the "required" keyword.  If a schema defines a set of properties that are guaranteed to exist, then other schemas that inherit from this (using "allOf") can define more properties that are guaranteed to exist, but they cannot make the original properties non-compulsory.

So I think we've been talking at cross-purposes here.  The set of methods/properties that are guaranteed to exist grows (defines a superset) when you inherit, in both OO and JSON Schema.  However, this creates a subset of possible values (in both OO and JSON Schema) because you are no longer allowed objects that don't define these properties.

I think that OO and JSON Schema are currently in agreement when it comes to inheritance.

On Monday, January 21, 2013 5:30:07 PM UTC, Austin Wright wrote:

[Geraint (David)]

"Every instance of Super may be read into an instance of Sub"?

How would you de-serialize (3, 4) into a Sub?  To be a valid Sub, the third value *must* have a defined value.

Extending vector spaces does not work the same way as OO inheritance.  The problem being that when you extend a vector space, one of the (sometimes implicit) steps is defining a homomorphism from the extended space to the original.  Your homomorphism here seems to be truncation - but OO inheritance doesn't work like that.  If I simply truncate the segment of memory from an object of a sub-class and treat it as its super-class and call a method on it, it will not behave the same way as calling the method first and then truncating the data.  Your homomorphism is broken.

(Well, unless you're in C++ and for some reason not using virtual methods, at which point they are equivalent.  But that is most definitely the exception and not the rule.)

[Austin Wright]

On Monday, January 21, 2013 11:25:24 AM UTC-7, Geraint (David) wrote:

"Every instance of Super may be read into an instance of Sub"?

How would you de-serialize (3, 4) into a Sub?  To be a valid Sub, the third value *must* have a defined value.

When you initialize Sub, the all three variables are undefined (unless you're using a programming language that initializes variables for you). Then when you read (3, 4) into Sub, the third variable simply remains undefined (or default value).

[Jim Klo]

Or to put it another way, there exists a serialized data structure that is a valid Sub, but not a valid Super, but no such structure the other way around. Every instance of Super may be read into an instance of Sub, but an instance of Sub can not be unserialized and read into a Super instance.

Actually that's not quite 100% accurate. An instance of Sub can be fully unserialized into a Super, however not quite the reverse..

Consider... All Ferraris are Cars, but not all Cars are Ferraris.  The subclass is always a specialization of the superclass.  Hence you can serialize a Ferrari into a Car... but you can only partially marshall a Car into a Ferrari...

[Jim Klo]

So after this rather lengthy discussion... I'm a bit still confused... 

This is what I got out of this:

    "extends" doesn't really do inheritance... and everyone hates the term.

    "additionalProperties": false is somewhat like the Java "final" keyword on a class.

    "type" sort of does the same thing as "extends" but is really a union or conjunction (almost as bad a "friends" in C++).

    all of this is reworked in v4 with a bit more clarity.

Now following the recommendation from another thread, from fge I believe, I did something like this:

foo.json

{

  "id": "http://example.com/foo.json",

  "properties": {

        "foo": {

            "type": "string"

         }

   }

}

bar.json

{

  "id": "http://example.com/bar.json",

  "extends": {

    "$ref": "http://example.com/foo.json"

   },

  "properties": {

    "bar": { "type": "string" }

  },

  "additionalProperties": false

}

Should this permit a instance of bar like { "foo": "hello", "bar": "world" } but not { "foo": "hello", "bar": "world", "name": "bob" }?  That's what I want to do... and would expect..

The validator, I'm using (python jsonschema https://github.com/Julian/jsonschema) doesn't seem to honor the the inherited properties.. If there's a better python solution I'm open to change. However I'm trying to determine if this is a bug or feature of v3. From what the dialoge suggests it seems like a feature, for the reason that bar can't apply the restriction to foo because of a 'lack of visibility to foo', which I'd call BS on since there's a reference right there to foo in bar.

[Geraint (David)]

Actually - I think "extends" absolutely does do classical inheritance.  People were confused by it, so it was renamed to "allOf", but it is indeed inheritance.

"type" is version 3 - with the latest version of the standard you should use either "anyOf" (like an OR operator) or "oneOf" (like an XOR operator) depending on what you need.

On Tuesday, January 22, 2013 8:08:16 PM UTC, Jim Klo wrote:

So after this rather lengthy discussion... I'm a bit still confused... 

This is what I got out of this:

    "extends" doesn't really do inheritance... and everyone hates the term.

    "additionalProperties": false is somewhat like the Java "final" keyword on a class.

    "type" sort of does the same thing as "extends" but is really a union or conjunction (almost as bad a "friends" in C++).

    all of this is reworked in v4 with a bit more clarity.

Now following the recommendation from another thread, from fge I believe, I did something like this:

foo.json

{

  "id": "http://example.com/foo.json",

  "properties": {

        "foo": {

            "type": "string"

         }

   }

}

bar.json

{

  "id": "http://example.com/bar.json",

  "extends": {

    "$ref": "http://example.com/foo.json"

   },

  "properties": {

    "bar": { "type": "string" }

  },

  "additionalProperties": false

}

Should this permit a instance of bar like { "foo": "hello", "bar": "world" } but not { "foo": "hello", "bar": "world", "name": "bob" }?  That's what I want to do... and would expect..

Actually, not - it will only permit instances of {"bar": "world"}.  The "additionalProperties" operator doesn't pay any attention to "extends", so it uses just the properties from "properties" (or "patternProperties" if present).  As such, your bar.json only permits the single property "bar".

 

The validator, I'm using (python jsonschema https://github.com/Julian/jsonschema) doesn't seem to honor the the inherited properties.. If there's a better python solution I'm open to change. However I'm trying to determine if this is a bug or feature of v3. From what the dialoge suggests it seems like a feature, for the reason that bar can't apply the restriction to foo because of a 'lack of visibility to foo', which I'd call BS on since there's a reference right there to foo in bar.

See above for how it works.

The reason it works like this is simple:  what if you have a schema:

base.json

{

    "type": "object",

    "properties": {

        "firstName": {"type": "string", ...},

        "middleName": {"type": "string", ...},

        "lastName": {"type": "string", ...}

    },

    "required": ["firstName", "lastName"]

}

and you want to restrict it to a stricter format - you want "firstName" and "lastName", but no other properties.  In this case, you can do this:

strict.json

{

    "allOf": [{"$ref": "base.json"}],

    "properties": {

        "firstName": {},

        "lastName": {}

    },

    "additionalProperties": false

}

[Jim Klo]

On Jan 22, 2013, at 12:50 PM, "Geraint (David)" <gerai...@gmail.com>

 wrote:

So then is the solution to my problem then the following? 

bar.json

{

  "id": "http://example.com/bar.json",

  "extends": {

    "$ref": "http://example.com/foo.json"

   },

  "properties": {

           "foo": {},

    "bar": { "type": "string" }

  },

  "additionalProperties": false

}

It's confusing because is "foo" being inherited or overridden? If inherited, then this should be equivalent?

bar.json

{

  "id": "http://example.com/bar.json",

  "properties": {

           "foo": {

    "$ref": "http://example.com/foo.json#properties/foo"

      },

    "bar": { "type": "string" }

  },

  "additionalProperties": false

}

 

The validator, I'm using (python jsonschema https://github.com/Julian/jsonschema) doesn't seem to honor the the inherited properties.. If there's a better python solution I'm open to change. However I'm trying to determine if this is a bug or feature of v3. From what the dialoge suggests it seems like a feature, for the reason that bar can't apply the restriction to foo because of a 'lack of visibility to foo', which I'd call BS on since there's a reference right there to foo in bar.

See above for how it works.

The reason it works like this is simple:  what if you have a schema:

base.json

{

    "type": "object",

    "properties": {

        "firstName": {"type": "string", ...},

        "middleName": {"type": "string", ...},

        "lastName": {"type": "string", ...}

    },

    "required": ["firstName", "lastName"]

}

and you want to restrict it to a stricter format - you want "firstName" and "lastName", but no other properties.  In this case, you can do this:

strict.json

{

    "allOf": [{"$ref": "base.json"}],

    "properties": {

        "firstName": {},

        "lastName": {}

    },

    "additionalProperties": false

}

Okay, that for some reason just seems terrible. That's just bad OO design - IFF JSON Schema is intended to be applied in an OO manner… In strict OO inheritance you can't subclass to "hide"  properties… Your example highlights something more like a Java interface than inheritance (i.e. strict.json is not a base.json)  And if that's the intent.. I'll accept it but it's not really more polymorphism (via an interface) than inheritance because your changing the access to base.json. Technically you can't remove the property via inheritance, you can only override or mask the underlying value.

FWIW: Here's a excellent refresher on inheritance, interfaces, and polymorphism http://www3.ntu.edu.sg/home/ehchua/programming/java/J3b_OOPInheritancePolymorphism.html

[Geraint (David)]

Yes, that is the solution.

"foo.json" is being inherited.  The constraints you place in one schema do not override the constraints in another - they join together.

The reason you need to specify the property "foo" in "bar.json" is because you're not just defining a new property - if that were the case, you could just define "bar" and leave it.  The difference is that you are adding the new constraint "additionalProperties": false.  That constraint uses the list of keys in "properties"/"patternProperties" to determine what it does.

Here, again, we have the difference between defined properties, allowed properties, and required properties:

In OO, when you define a method/property on a class, that property is required for all instances of the class.  It MUST be defined - you cannot have an instance that does not define one of the methods, for example.  And you are absolutely right - in OO inheritance, "hiding" a property would mean it's no longer required, so the code cannot use it, because it is not guaranteed to exist.  So sub-classes cannot "hide" properties/methods, they can only define new ones on top.

This is directly equivalent to the "required" keyword in JSON Schema.  If your base schema specifies certain properties in "required", then a schema that inherits from that cannot "hide" those properties - it can only add to the set of required properties.

But JSON Schema has the idea of properties that are defined, but still optional.  This can't be compared to classical OO behaviour, because classical OO has no concept of optional properties/methods.  The closest you come is the idea of a method that might return null, or something.  It is one of these optional properties ("middleName") that the above example "hides".

[Jim Klo]

Okay... all makes sense...

so I have a seemingly related issue, wanting write tests property by property since there are many many combinations and I don't really want to clutter the tests with mounds of data samples.

so I have this base schema like so (abbreviated the property list for clarity... imagine another 20 - 30 properties):

{

    "$schema": "http://tools.ietf.org/html/draft-zyp-json-schema-03",

    "title": "abstract_resource_data",

    "description": "Abstract Resource Data",

    "properties": {

        "doc_type": {

            "type": "string",

            "enum": ["resource_data"],

            "required": true

        }

    },

    "patternProperties" : {

        "X_.*": { "type":"any" }

    }

}

in my test... i'm using the following test case to test my schema property by property:

[

    {

        "description": "validate doc_type",

        "schema":{

            "type": "object",

            "properties": {

                "doc_type": { "$ref": "file:lr/schema/abstract_resource_data.json#properties/doc_type" }

            }

        },

        "tests":[

            {

                "description": "good doc_type",

                "data": { "doc_type": "resource_data" },

                "valid": true

            },

            {

                "description": "bad doc_type",

                "data": { "doc_type": "activity_data" },

                "valid": false

            },

            {

                "description": "missing doc_type",

                "data": {  },

                "valid": false

            }

        ]

    }

]

What I don't understand is that the referenced schema from the test has "required": true.... shouldn't this get caught in the "missing doc_type" test?  I've tried this with a number of different permutations as discussed in this thread with no luck...  I either get into the situation of the constraint not being enforced... or all of the required inherited properties needing to be enforced...

does "required":true not get inherited when referenced directory from the property?  Is there a way to solve this with V3 or am I SOL?

[Francis Galiegue]

Hello,

On Fri, Jan 25, 2013 at 4:45 AM, Jim Klo <jim...@sri.com> wrote:
[snip]

> "description": "Abstract Resource Data",
> "properties": {
> "doc_type": {
> "type": "string",
> "enum": ["resource_data"],
> "required": true
> }
> },

[snip]

> "patternProperties" : {
> "X_.*": { "type":"any" }

Warning: regexes in JSON Schema are not anchored. If you want to match
properties starting with "X_", write "^X_." as a regex, no need for
more.

The way this regex is written, it will match, for instance, "AX_"
(since ".*" can match the empty string)

>
> in my test... i'm using the following test case to test my schema property
> by property:
>
> [
> {
> "description": "validate doc_type",
> "schema":{
> "type": "object",
> "properties": {
> "doc_type": { "$ref":
> "file:lr/schema/abstract_resource_data.json#properties/doc_type" }
> }

[snip]

>
> What I don't understand is that the referenced schema from the test has
> "required": true.... shouldn't this get caught in the "missing doc_type"
> test?

What validator are you using? The behaviour you see here is
unfortunately normal, and is why "required" has been defined as such
in draft v4.

Look at your schema above:

> "properties": {
> "doc_type": {
> "type": "string",
> "enum": ["resource_data"],
> "required": true
> }

If you extract the schema you reference, this gives:

{

"type": "string",
"enum": [ "resource_data" ],
"required": true
}

(by the way: get rid of "type": "string", you don't need it here --
the constraint on "enum" is enough by itself)

And this schema will be evaluated _in its own context_. And in this
context, "required" means nothing. Therefore, the constraint cannot be
enforced.

--
Francis Galiegue, fgal...@gmail.com
Try out your JSON Schemas: http://json-schema-validator.herokuapp.com

[Jim Klo]

On Jan 24, 2013, at 8:04 PM, Francis Galiegue <fgal...@gmail.com>

 wrote:

Hello,

On Fri, Jan 25, 2013 at 4:45 AM, Jim Klo <jim...@sri.com> wrote:
[snip]

   "description": "Abstract Resource Data",
   "properties": {
       "doc_type": {
           "type": "string",
           "enum": ["resource_data"],
           "required": true
       }
   },

[snip]

   "patternProperties" : {
       "X_.*": { "type":"any" }

Warning: regexes in JSON Schema are not anchored. If you want to match
properties starting with "X_", write "^X_." as a regex, no need for
more.

The way this regex is written, it will match, for instance, "AX_"
(since ".*" can match the empty string)

Thanks… good catch...

in my test... i'm using the following test case to test my schema property
by property:

[
   {
       "description": "validate doc_type",
       "schema":{
           "type": "object",
           "properties": {
               "doc_type": { "$ref":
"file:lr/schema/abstract_resource_data.json#properties/doc_type" }
           }

[snip]

What I don't understand is that the referenced schema from the test has
"required": true.... shouldn't this get caught in the "missing doc_type"
test?

What validator are you using? The behaviour you see here is
unfortunately normal, and is why "required" has been defined as such
in draft v4.

Using https://github.com/Julian/jsonschema/, mostly because it has the most complete set of unit tests… 

However I've extended the Draft3Validator to handle "format" and the RefResolver to handle relative file: paths in a way that urlopen understands; my extension is here: https://gist.github.com/4631731

Makes sense now why "required" in v4 is drafted that way…  If there was a v4 validator that I didn't have to completely author… in python… I'd be golden... :)

Look at your schema above:

   "properties": {
       "doc_type": {
           "type": "string",
           "enum": ["resource_data"],
           "required": true
       }

If you extract the schema you reference, this gives:

{
   "type": "string",
   "enum": [ "resource_data" ],
   "required": true
}

(by the way: get rid of "type": "string", you don't need it here --
the constraint on "enum" is enough by itself)

And this schema will be evaluated _in its own context_. And in this
context, "required" means nothing. Therefore, the constraint cannot be
enforced.

I think that's slowly sinking in… :) 

So the solution is… find or build a v4 validator or test "required" as a a series of minimally required objects with properties… or make dozens of schemas for each property (bleech).

Thanks,...