$ref, $merge, $extend (for object oriented data model generation)

[Simon Heimler]

Hello!

I've already mentioned this problem as a sidenote on this mailling-list, but now that I've implemented and used it, I'd like to throw it into discussion.

In Order to have a DRY model, I was re-using JSON Schemas a lot. This can be done via the $ref attribute, but the official JSON Schema Specification does not define if there is inheritance behaviour. (and if, how).

Inheritance behaviour made a lot of sense to me, since I can inherit attributes and still overwrite them. In my own (prototype) implementation I've introduced a $extend attribute that does exactly that: It makes a deep copy of the current scope of the $extend copy and merges it with the referenced $extend object.

I'll make an example:

An abstract Shape Model:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "title": "Shape",
    "description": "Generic Shape",
    "type": "object",
    "properties": {
        "x": {
            "type": "integer"
        },
        "y": {
            "type": "integer"
        }
    },
    "required": ["x", "y"],
    "abstract": true
}

Then there's a Circle, that extends from Shape:

{
    "$extend": "/model/_Shape.json",
    "title": "Circle",
    "type": "object",
    "properties": {
        "radius": { "$extend": "/field/radius.json" },
    },
    "required": ["x", "y", "radius"],
    "abstract": false
}

So, the properties x and y are inherited. The title is overwritten. The property radius is also $extended, so properties can be reused between different models.

I hope you get the idea.

So here's my question: Would it make sense to have the spec define if and how inheritance should be implemented? I've noticed that some implementations (I think it was t4v) was actually doing inheritance with $ref. But this is nothing the spec talks about, so I can't count that this behaviour is consistent through different implementations. Since I wasn't doing something official, I decided to implement my own $extend attribute and the inheritance logic behind it. 

And there's the naming thing. I also read of a $merge attribute, but didn't found much informations about it.

Feel free to tell me what you think of it, if that makes sense to you.

If you're insterested in the background of this, I've created a project called "mobo" that uses JSON Schema for Graph Data Model Generation in Semantic MediaWiki. The model is written in JSON Schema and the Model, the Forms, Documention and through Semantic MediaWiki even Semantic Attributes are automatically generated. Here's a presentation about it: http://fannon.de/p/mobo-intro/#/

Kind Regards,

Simon

[Christopher J. White]

Hi Simon,

This looks very similar to $merge, just a slightly different syntax.

In the $merge case you have:

  $merge:
     source: <schema>
     with: <schema>

Where the two schemas can be anything -- direct inline schemas, or both $refs that get resolved to other schemas.  As such $merge is a more flexible and it seems $extend.

  $extend: <reference>
  <modified attributes>

This would be expressed with $merge as:

  $merge:
    source: { $ref: <reference> }
    with:
        <modified attributes>

Personally I like this simplified syntax, because in I think every use case we have for $merge, it is of this form where source is a $ref and one or more attributes are changed or added. 

One question:

    "properties": {
        "radius": { "$extend": "/field/radius.json" },
    },

It doesn't seem like you are actually extending radius -- wouldn't this just be a $ref?

...cj

[Simon Heimler]

Thanks for explaining this! This is pretty straightforward. And I understand the case that it might happen to merge two external $refs, this can't be done with $extend. But what happens if there are three or more schemas to be merged? In that case I'm using the "allOf" attribute and use $extend several times. In that array the order counts and the last elements overwrite the first. One thing that I like more with $extend than $merge is that it doesn't introduce yet another nested level of an object. 

Your Question: Yes, the radius is just a $ref, but I dont wanted to mix those concepts and $extend does in that case exactly the same + potentially more. 

And given the case I later decide that I want to overwrite something from the radius attribute, I just add the modified attributes after the $extend attribute, no need to change the current structure or change the attribute afterwards:

"properties": {
    "radius": { 
        "$extend": "/field/radius.json",

        "minimum": 5
    },
},

Of course I'm not going the standardized way here and I gladly would - if this would be possible with v5 for example. But for that to happen the inheritance / merge behaviour has to be defined. Of course the way you explain $merge this would work just fine, with the slight disadvantage of me to change to a little more verbose notation. 

Greets,

Simon 

--
You received this message because you are subscribed to a topic in the Google Groups "JSON Schema" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/json-schema/anDZaPeksMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to json-schema...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simon Heimler]

For better explanation I'll give you an example how multiple $extends could look like:

{
    "allOf": [
        {"$extend": "/model/_Object.json"},
        {"$extend": "/model/_Shape.json"},
        {"$extend": "/model/_Movable.json"}
    ],

    "title": "Circle",
    "type": "object",

    "properties": {
        "radius": { "$extend": "/field/radius.json" },
    },

    "required": ["x", "y", "radius"],
    "abstract": false
}

Now that I have thought of it, the $extend notation has one more benefit for me: If there's only one $extend attribute its on the same level with the attributes that are do overwrite the model. After the $extend is "calculated/done", all those attributes stay on the level they are currently in. With $merge an additional layer is introduced and after the calculation thrown away.

Just my thoughts on that

Simon

[Christopher J. White]

Hmm -- I'm not sure about this use of allOf.  I understand what you're getting at, but seems to me that in addition to adding a new keyword ($extend), you have also modified the meaning of allOf to know about $extend and to special processing in it's presence.  With the classic v4 definition of allOf, the array of schemas are wholly independent schemas that must be satisfied independently.  You're use seems to break that because if Shape overrides a property that was defined in Object, then the Object schema may not be expected to validate properly.

With $merge, you would get the desired behavior via multiple levels, which while not pretty does work:

$merge:
   source:
      $merge:
         source: { $ref: Object }
         with: { $ref: Shape }
   with: { $ref: Movable }

The added layer of nesting with $merge is indeed the downside.  

Currently $ref is defined as a separate draft that was written independent of JSON Schema and is long since expired.  My understanding is that the intention is to bring $ref into the fold of JSON Schema.  The current (expired) definition of $ref specifically indicates that other properties alongside the $ref shall be ignored.  I would be in favor or relaxing that so that we could just use $ref in the way you are using $extend.

  $ref: Object
  title: "Circle"

In the event that you want multiple cascaded definitions, maybe $ref can take an array:

  $ref: [ Object, Shape, Movable ]
  title: "Circle"

...cj

You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema...@googlegroups.com.

[Dimitar Draganov]

Hi Simon,

while I'm also quite fond of DRY principle and for same reason in favor of '$merge' being adopted as part of v5, your shape/circle sample is quite easy expressed by 'allOf' keyword. I personally think that combination between 'allOf' and '$merge' can cover almost all inheritance cases. What I did however in my own implementation was to not only pre-process '$ref', but also merge 'allOf' schemas for visualization purpose only, while validation stays intact. This way I can really use 'allOf' exactly as you describe it in your later sample, where one schema is build by 'including' properties from many other base schemas.

As it goes for 'abstract' I must admit I do have a case for it, since I also define many schemas to only serve as 'base' and be used exclusively in 'allOf', however I still consider two main goals of json-schema to be description of instance object and its validation. So having 'sealed' schemas is not really needed to achieve those goals and I can easy live without it.

One thing I have found against using '$merge' for inheritance is the fact that you can actually 'break' base schema and make instance which is valid against child schema to be not valid against base one, this can be permitted in some cases and undesired in other.

Regards, Dimitar

[Simon Heimler]

Thanks for your replies :)

Now I'm a bit confused regarding the use of "allOf". But you might be right Christopher, I wasn't sure if it was meant to be used this way. And now I'm still not sure after @Dimitars explanation ;)

But If the $ref spec would be updated like you said, this would be the drop-in replacement for my unofficial solution. I do like the array solution, you mentioned - its clean, simple and doesn't require mixing of different concepts / keywords : $ref: [ Object, Shape, Movable ]

@Dimitar: I've got no issues with that validation because I always transform/extend my JSON Schema first (i call that process "deepening"). Then I end up with completely usual JSON Schema files where every $extend is replaced/calculated. So I'm using $extend just for DRY JSON Schema Development. In the next step standard conform JSON Schema is generated which isn't dry anymore, but simple to use for my purposes. This is somewhat similar to the JSON-LD expand process.

One question about the naming of "$merge" - this somewhat confused me first. 

If I hear merge I'm thinking about a two-directional process where changes from both sides are incorporated (Version Control) - rather like syncing. That was a reason for me to use "$extend", since there the direction (Child $extends Father) is clear. I know that the underscore library calls the same process merging, but I had some difficulties remembering the direction and it was somewhat inuntuitive to me.

Regards,

Simon

--

[Christopher J. White]

On 10/9/14, 1:18 AM, Simon Heimler wrote:

@Dimitar: I've got no issues with that validation because I always transform/extend my JSON Schema first (i call that process "deepening"). Then I end up with completely usual JSON Schema files where every $extend is replaced/calculated. So I'm using $extend just for DRY JSON Schema Development. In the next step standard conform JSON Schema is generated which isn't dry anymore, but simple to use for my purposes. This is somewhat similar to the JSON-LD expand process.

I'll note that for both $ref and $merge, I've found it necessary to perform lazy evaluation of such references, and IIRC some libraries out there that support $ref do the same.  This is because it is possible to generate circular references that are perfectly acceptable. 

Consider defining a JSON schema for a binary tree:

  id: tree
  type: object
  properties:
    name: { type: string }
    left:
      oneOf:
         - type: null
         - $ref: "#tree"
    right:
      oneOf:
         - type: null
         - $ref: "#tree"

Endless recursion if you try to replace all $refs at run time!

One question about the naming of "$merge" - this somewhat confused me first. 

If I hear merge I'm thinking about a two-directional process where changes from both sides are incorporated (Version Control) - rather like syncing. That was a reason for me to use "$extend", since there the direction (Child $extends Father) is clear. I know that the underscore library calls the same process merging, but I had some difficulties remembering the direction and it was somewhat inuntuitive to me.

One piece that I sort of glossed over is that the $merge operates at the JSON layer, *not* JSON Schemas.  This is important because it allows you to pass in parts for "source" and "with" that are otherwise not valid JSON Schemas.

For example:

  $merge:
    source:
      type: integer
    with:
      mininum: 5

The latter "with" clause is not a schema by itself because "minimum" only applies to integers, yet it has no type.

As such, it's more of a merge than extension, where precedence for conflicts is always on the "with" clause.

...cj
     

Regards,

Simon

2014-10-08 20:54 GMT+02:00 Dimitar Draganov <ddman...@gmail.com>:

Hi Simon,

while I'm also quite fond of DRY principle and for same reason in favor of '$merge' being adopted as part of v5, your shape/circle sample is quite easy expressed by 'allOf' keyword. I personally think that combination between 'allOf' and '$merge' can cover almost all inheritance cases. What I did however in my own implementation was to not only pre-process '$ref', but also merge 'allOf' schemas for visualization purpose only, while validation stays intact. This way I can really use 'allOf' exactly as you describe it in your later sample, where one schema is build by 'including' properties from many other base schemas.

As it goes for 'abstract' I must admit I do have a case for it, since I also define many schemas to only serve as 'base' and be used exclusively in 'allOf', however I still consider two main goals of json-schema to be description of instance object and its validation. So having 'sealed' schemas is not really needed to achieve those goals and I can easy live without it.

One thing I have found against using '$merge' for inheritance is the fact that you can actually 'break' base schema and make instance which is valid against child schema to be not valid against base one, this can be permitted in some cases and undesired in other.

Regards, Dimitar

--
You received this message because you are subscribed to a topic in the Google Groups "JSON Schema" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/json-schema/anDZaPeksMA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to json-schema...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "JSON Schema" group.

To unsubscribe from this group and stop receiving emails from it, send an email to json-schema...@googlegroups.com.

[Simon Heimler]

​

One piece that I sort of glossed over is that the $merge operates at the JSON layer, *not* JSON Schemas.  This is important because it allows you to pass in parts for "source" and "with" that are otherwise not valid JSON Schemas.​

Well, thats definately an argument for calling it a $merge ;)

So to sum it up for me:

• $merge looks like a great feature for v5 to me, really looking forward to it!
  
• If the $ref spec is updated I would prefer if it works through inheritance, but if not it wouldn't be a big problem since $merge can do that anyway.
  
• The use of "allOf" together with $ref or $merge should be somewhere explained in detail
• Right now I do stick with my $extend solution since the spec v5 is not released. If the $ref gets inheritance it would be a simple drop in replacement.

Thanks for the explanations!

Regards,

Simon