allOf / additionalProperties conundrum
290 views
Skip to first unread message
Elvis Stansvik
Dec 15, 2016, 3:25:44 AM12/15/16
to json-...@googlegroups.com
Hi all,
I've written a schema that I'm quite happy with. You can see it here
in YAML syntax:
https://gist.github.com/estan/a8f824d657c07203df4641610cae2aed
It's the first time I use JSON Schema, and I must say I really like it.
There's one thing that my schema really miss though, and that's
disallowing additional properties in most places.
The reason I haven't added this is because I'm using a construct like
allOf:
- type: object
properties:
# Properties common to all items
anyOf:
- type: object
properties:
# Properties for one type of item
- type: object
properties:
# Properties for another type of item
to express an is-a relationship (see the "specification" property in
the schema), which AFAICS makes it impossible to use
additionalProperties: false, since the "reach" of the
additionalProperties only extends to the lexical scope so to speak.
I've seen others having the same problem, but I've never seen any
satisfactory solution, so I thought I'd ask you experts: Is there any
way to solve this without going through hoops or violating DRY?
Suggestions I've seen are:
1. Simply live with repetition and ditch the allOf, and repeat
common properties in each item.
2. Simply give up and allow additional properties
3. Various restructurings of the schema that makes it less legible
and still having some repetition
I don't like 1 because it makes editing the schema error prone.
Number 2 would be an option, and normally I think I'd do this, but in
this case I really want to disallow additional properties. The reason
is this: The schema describes a specification for a file/tree
structure. These specifications are used as input for a validation
tool we use to validate data format of ours. If additional properties
are allowed, it's very easy for someone to change something in the
validation tool and forgetting to update all our specs. Data could
then still seemingly validate, but in fact be invalid, which would be
very bad.
Regarding number 3, I still haven't found a solution that expresses
what I mean as clearly as the above, that's kind of why I'm asking
here if anyone has found a good solution to this problem.
I know that v5 introduces some things that would solve this (merge?)
elegantly, but our tooling uses PyYAML, so I must stick to v4.
Many thanks in advance!
Elvis
I've written a schema that I'm quite happy with. You can see it here
in YAML syntax:
https://gist.github.com/estan/a8f824d657c07203df4641610cae2aed
It's the first time I use JSON Schema, and I must say I really like it.
There's one thing that my schema really miss though, and that's
disallowing additional properties in most places.
The reason I haven't added this is because I'm using a construct like
allOf:
- type: object
properties:
# Properties common to all items
anyOf:
- type: object
properties:
# Properties for one type of item
- type: object
properties:
# Properties for another type of item
to express an is-a relationship (see the "specification" property in
the schema), which AFAICS makes it impossible to use
additionalProperties: false, since the "reach" of the
additionalProperties only extends to the lexical scope so to speak.
I've seen others having the same problem, but I've never seen any
satisfactory solution, so I thought I'd ask you experts: Is there any
way to solve this without going through hoops or violating DRY?
Suggestions I've seen are:
1. Simply live with repetition and ditch the allOf, and repeat
common properties in each item.
2. Simply give up and allow additional properties
3. Various restructurings of the schema that makes it less legible
and still having some repetition
I don't like 1 because it makes editing the schema error prone.
Number 2 would be an option, and normally I think I'd do this, but in
this case I really want to disallow additional properties. The reason
is this: The schema describes a specification for a file/tree
structure. These specifications are used as input for a validation
tool we use to validate data format of ours. If additional properties
are allowed, it's very easy for someone to change something in the
validation tool and forgetting to update all our specs. Data could
then still seemingly validate, but in fact be invalid, which would be
very bad.
Regarding number 3, I still haven't found a solution that expresses
what I mean as clearly as the above, that's kind of why I'm asking
here if anyone has found a good solution to this problem.
I know that v5 introduces some things that would solve this (merge?)
elegantly, but our tooling uses PyYAML, so I must stick to v4.
Many thanks in advance!
Elvis
Henry Andrews
Dec 15, 2016, 3:48:08 AM12/15/16
to json-...@googlegroups.com
Hi Elvis,
There's a GitHub issue that addresses how you could write a tool to handle this: https://github.com/json-schema-org/json-schema-spec/issues/119
The "$combine"/"$combinable" proposal is very unlikely to go directly into JSON Schema itself, but it could be done as a pre-processing step. The approach is something like your approach #3, I think.
cheers,
-henry
From: Elvis Stansvik <elvs...@gmail.com>
To: json-...@googlegroups.com
Sent: Thursday, December 15, 2016 12:25 AM
Subject: [json-schema] allOf / additionalProperties conundrum
--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema+unsub...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Elvis Stansvik
Dec 15, 2016, 4:21:40 AM12/15/16
to json-...@googlegroups.com
2016-12-15 9:45 GMT+01:00 'Henry Andrews' via JSON Schema
<json-...@googlegroups.com>:
I didn't think of writing a pre-processing step. That is of course an
option, and the example $combine -> allOf transformation for the
simple case in the proposal looks very much like what I did manually
to my schema while experimenting. I didn't like the result because of
the repetitive "placeholder" properties I had to put in, but doing it
as a pre-processing step would hide the ugliness.
However, looking at what it would take to make this transformation
work with boolean operators (allOf, anyOf, oneOf and not) inside the
$combined schemas, I'm a bit hesitant. It seems non-trivial to get the
transformation right :/
I might consider making a half-baked such pre-processing step without
support for that (since I don't have any such constructs currently)...
...but, is there really no convenient way to express an is-a
relationship with additional properties prohibited, using only JSON
Schema v4? It seems like it would be a quite common use case.
Elvis
> email to json-schema...@googlegroups.com.
<json-...@googlegroups.com>:
> Hi Elvis,
> There's a GitHub issue that addresses how you could write a tool to handle
> this: https://github.com/json-schema-org/json-schema-spec/issues/119
> The "$combine"/"$combinable" proposal is very unlikely to go directly into
> JSON Schema itself, but it could be done as a pre-processing step. The
> approach is something like your approach #3, I think.
Thanks a lot for the pointer Henry,
> There's a GitHub issue that addresses how you could write a tool to handle
> this: https://github.com/json-schema-org/json-schema-spec/issues/119
> The "$combine"/"$combinable" proposal is very unlikely to go directly into
> JSON Schema itself, but it could be done as a pre-processing step. The
> approach is something like your approach #3, I think.
I didn't think of writing a pre-processing step. That is of course an
option, and the example $combine -> allOf transformation for the
simple case in the proposal looks very much like what I did manually
to my schema while experimenting. I didn't like the result because of
the repetitive "placeholder" properties I had to put in, but doing it
as a pre-processing step would hide the ugliness.
However, looking at what it would take to make this transformation
work with boolean operators (allOf, anyOf, oneOf and not) inside the
$combined schemas, I'm a bit hesitant. It seems non-trivial to get the
transformation right :/
I might consider making a half-baked such pre-processing step without
support for that (since I don't have any such constructs currently)...
...but, is there really no convenient way to express an is-a
relationship with additional properties prohibited, using only JSON
Schema v4? It seems like it would be a quite common use case.
Elvis
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema...@googlegroups.com.
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Elvis Stansvik
Dec 15, 2016, 4:28:06 AM12/15/16
to json-...@googlegroups.com
just put the "common properties" I have in a definition and $ref them
in my "sub-items". But alas, the "requiredProperties: false" doesn't
even reach across $ref'ed properties AFAIK, so that's not an option :/
Elvis
Elvis Stansvik
Dec 15, 2016, 4:37:01 AM12/15/16
to json-...@googlegroups.com
2016-12-15 10:21 GMT+01:00 Elvis Stansvik <elvs...@gmail.com>:
> 2016-12-15 9:45 GMT+01:00 'Henry Andrews' via JSON Schema
> <json-...@googlegroups.com>:
>> Hi Elvis,
>> There's a GitHub issue that addresses how you could write a tool to handle
>> this: https://github.com/json-schema-org/json-schema-spec/issues/119
>> The "$combine"/"$combinable" proposal is very unlikely to go directly into
>> JSON Schema itself, but it could be done as a pre-processing step. The
>> approach is something like your approach #3, I think.
>
> Thanks a lot for the pointer Henry,
>
> I didn't think of writing a pre-processing step. That is of course an
> option, and the example $combine -> allOf transformation for the
> simple case in the proposal looks very much like what I did manually
> to my schema while experimenting. I didn't like the result because of
> the repetitive "placeholder" properties I had to put in, but doing it
> as a pre-processing step would hide the ugliness.
>
> However, looking at what it would take to make this transformation
> work with boolean operators (allOf, anyOf, oneOf and not) inside the
> $combined schemas, I'm a bit hesitant. It seems non-trivial to get the
> transformation right :/
>
> I might consider making a half-baked such pre-processing step without
> support for that (since I don't have any such constructs currently)...
And even if I did this, I think I would also have to re-implement the
> <json-...@googlegroups.com>:
>> Hi Elvis,
>> There's a GitHub issue that addresses how you could write a tool to handle
>> this: https://github.com/json-schema-org/json-schema-spec/issues/119
>> The "$combine"/"$combinable" proposal is very unlikely to go directly into
>> JSON Schema itself, but it could be done as a pre-processing step. The
>> approach is something like your approach #3, I think.
>
> Thanks a lot for the pointer Henry,
>
> I didn't think of writing a pre-processing step. That is of course an
> option, and the example $combine -> allOf transformation for the
> simple case in the proposal looks very much like what I did manually
> to my schema while experimenting. I didn't like the result because of
> the repetitive "placeholder" properties I had to put in, but doing it
> as a pre-processing step would hide the ugliness.
>
> However, looking at what it would take to make this transformation
> work with boolean operators (allOf, anyOf, oneOf and not) inside the
> $combined schemas, I'm a bit hesitant. It seems non-trivial to get the
> transformation right :/
>
> I might consider making a half-baked such pre-processing step without
> support for that (since I don't have any such constructs currently)...
$ref mechanism, because I wouldn't want to do
anyOf:
- $combine:
- type: object
properties:
# Common properties
additionalProperties: false
- type: object
# Properties for one type of item
additionalProperties: false
- $combine:
- type: object
properties:
# Common properties
additionalProperties: false
- type: object
# Properties for another type of item
additionalProperties: false
but rather
anyOf:
- $combine:
- $ref: '#/definitions/common_properties'
- type: object
# Properties for one type of item
additionalProperties: false
- $combine:
- $ref: '#/definitions/common_properties'
- type: object
# Properties for another type of item
additionalProperties: false
definitions:
common_properties:
# Common properties
additionalProperties: false
so the pre-processing step would need support for $ref.
Elvis
Henry Andrews
Dec 15, 2016, 2:23:47 PM12/15/16
to json-...@googlegroups.com
There is no way to do this in draft 4. As far as I can tell, the bitter disagreements between the two principle authors of draft 4 on how to address this were what caused the project to be abandoned for three years after that draft.
Yes, implementing $combine fully is extremely complex- that's why even though I wrote the proposal I don't support it (I filed it to get someone else to stop complaining about a lack of alternate proposals for solving it).
And yes, $combine + $ref is a bit tricky and not strictly a preprocessor- it would have to be more of a late-binding just-in-time sort of thing.
Mostly, I avoid "additionalProperties": false, although that has a lot to do with how I am using JSON Schema for APIs and need to support evolvability and compatibility. Checking for unexpected properties is something I might enable during a testing cycle, but would not put it in the schema for production.
thanks,
-henry
Sent: Thursday, December 15, 2016 1:21 AM
Subject: Re: [json-schema] allOf / additionalProperties conundrum
Subject: Re: [json-schema] allOf / additionalProperties conundrum
> email to json-schema+unsub...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema+unsub...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema+unsub...@googlegroups.com.
Henry H. Andrews
Dec 29, 2016, 1:48:06 AM12/29/16
to JSON Schema
Hi Elvis,
See what you think of this idea using the newly added "propertyNames" feature:
thanks,
-henry
Elvis Stansvik
Dec 30, 2016, 8:03:32 AM12/30/16
to json-...@googlegroups.com
Hi Henry, sorry for the late reply.
>
> Yes, implementing $combine fully is extremely complex- that's why even
> though I wrote the proposal I don't support it (I filed it to get someone
> else to stop complaining about a lack of alternate proposals for solving
> it).
I see.
>
> And yes, $combine + $ref is a bit tricky and not strictly a preprocessor- it
> would have to be more of a late-binding just-in-time sort of thing.
Right.
>
> Mostly, I avoid "additionalProperties": false, although that has a lot to do
> with how I am using JSON Schema for APIs and need to support evolvability
> and compatibility. Checking for unexpected properties is something I might
> enable during a testing cycle, but would not put it in the schema for
> production.
I would also avoid it for that use case. But for what I'm working on
now, it would be very good for catching typo's, or catching the
mistake of updating the validation tool without also updating our
specifications.
Elvis
>> email to json-schema...@googlegroups.com.
> There is no way to do this in draft 4. As far as I can tell, the bitter
> disagreements between the two principle authors of draft 4 on how to address
> this were what caused the project to be abandoned for three years after that
> draft.
Aha. Did not know that.
> disagreements between the two principle authors of draft 4 on how to address
> this were what caused the project to be abandoned for three years after that
> draft.
>
> Yes, implementing $combine fully is extremely complex- that's why even
> though I wrote the proposal I don't support it (I filed it to get someone
> else to stop complaining about a lack of alternate proposals for solving
> it).
>
> And yes, $combine + $ref is a bit tricky and not strictly a preprocessor- it
> would have to be more of a late-binding just-in-time sort of thing.
>
> Mostly, I avoid "additionalProperties": false, although that has a lot to do
> with how I am using JSON Schema for APIs and need to support evolvability
> and compatibility. Checking for unexpected properties is something I might
> enable during a testing cycle, but would not put it in the schema for
> production.
now, it would be very good for catching typo's, or catching the
mistake of updating the validation tool without also updating our
specifications.
Elvis
>> For more options, visit https://groups.google.com/d/optout.
>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "JSON Schema" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to json-schema...@googlegroups.com.
>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "JSON Schema" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema...@googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema...@googlegroups.com.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
Henry Andrews
Dec 30, 2016, 12:20:36 PM12/30/16
to json-...@googlegroups.com
Take a look at https://github.com/json-schema-org/json-schema-spec/issues/214 for a way to catch properties that are not covered by combinations of schemas in Draft 6 (it requires the "propertyNames" keyword). It's a bit complex as you need to track your property schemas and your lists of valid names in parallel, but it's much less complex than "$combine" and has much more predictable results (because you don't have to worry about how it impacts nested combinations or negated schemas).
thanks,
-henry
Sent: Friday, December 30, 2016 5:03 AM
>> email to json-schema+unsub...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "JSON Schema" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to json-schema+unsub...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema+unsub...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "JSON Schema" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to json-schema+unsub...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages