Is it a good idea to have nested external $refs ? If it is, is there a JavaScript "aggregator" ?

[Simon Heimler]

Hello!

I've discovered JSON-Schema a few weeks ago and am very happy about that ;) Big Thanks!

Just one question about "best-practices" when using external $refs:

I've got some problems with Libraries when using nested $refs, meaning I've got a schema which references another, which references yet another.

The first external reference works just fine, but the 3rd does not. This may be, because nesting the refs leads to a recursive process and can result in bad performance.

In an actual project I'm not concerned about performance though and keeping the schema seperated in several dozend files would lead to much better maintainability. 

There could be one thing that would solve this problem for me: Having a (client side) JavaScript JSON "aggregator", which recursively loads and embeds all those external schema files and returns a complete Schema Object. https://github.com/natesilva/jayschema seems to support this, but it is a node.js application.

Is there some way to do what I want or am I on the wrong track with this idea?

Greets,

Simon

[Francis Galiegue]

Hello,

Well, my own implementation has no problem with that at all but you
don't tell which language you actually use. Is that JavaScript? If yes
my solution is not for you since it is pure Java...

--
Francis Galiegue, fgal...@gmail.com, https://github.com/fge
JSON Schema in Java: http://json-schema-validator.herokuapp.com
Parsers in pure Java: https://github.com/parboiled1/grappa (redde
Caesaris: https://github.com/sirthias)

[Simon Heimler]

Yes, i'm developing in client-side JavaScript ;)

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "JSON Schema" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/json-schema/fM46FTFwR-U/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> json-schema...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Eric G]

Hi Simon,

There's a number of javascript libraries listed at https://github.com/json-schema/JSON-Schema-Test-Suite. Probably the "standard" is https://github.com/geraintluff/tv4, by the author of the v4 draft, it does do remote dereferencing, and is browser-side.  My own set of libraries, https://github.com/ericgj/json-schema-suite, is designed to work in both node.js and browser, but is not very well maintained at the moment, and a bit of a pain to use, so I can't recommend it.  You might also check with the author of https://github.com/zaggino/z-schema how difficult it would be to adapt for browser use -- it's the newest implementation I know of and I believe being actively developed. 

HTH,
Eric

[Francis Galiegue]

Hello,

On Thu, May 15, 2014 at 3:34 AM, Eric G <eric...@gmail.com> wrote:
> Hi Simon,
>
> There's a number of javascript libraries listed at
> https://github.com/json-schema/JSON-Schema-Test-Suite. Probably the
> "standard" is https://github.com/geraintluff/tv4, by the author of the v4
> draft,
>

A little correction: David is the author of the hyperschema draft; I
am the author of the core and validation drafts ;)

[Simon Heimler]

Thanks for the reply's :)

I've played a bit around with the JavaScript Libraries and discovered that some of them support nestes $refs and some simply don't. A little bit of combing solved the problem for me.

This lead me to another idea: Schema Inheritance, instead of pure reference. 

My main goal right now is the development of a big and rather complex schema collection. In some cases it would be great (and DRY) to inherit a parent schema (or part of a schema) and add further properties to it in the child object.

This does already work to a specific point, using $ref and declaring further attributes below.

But if this is a good idea, maybe this is something to support officially? It could be nice, even for the standards itself, since json schema validation seems to inherit all properties from json-schema and extend it.

The implementation would not be difficult either, its just a matter of merging the referenced object with the object that is defined (without the $ref Attr. of course). Could be easily achived, with implementations like this: http://api.jquery.com/jQuery.extend/. In fact, it has only to be supported with the json-schema loader part of a library.

This would be a great benefit for developing big schemas. What is your oppinion on it?

(Should I open a GitHub Issue / Proposal for it and describe this idea further?)

Greets,

Simon

[Austin Wright]

It depends on what you want to do exactly. A while ago we were discussing instance inheritance - for instance, what does it mean to say "SchemaA subClassOf SchemaB"? An instance of SchemaA will also be a valid instnace of SchemaB, but what if I have additionalProperties set to false, or some custom value? What happens then?

I also recall proposing something similar, a "prototype" property iirc. I forget exactly what it did, but it sounds very similar. I'm not sure it was actually necessary, though, and the allOf property seems to cover most use-cases.

Provide some examples, maybe in a new message thread.

Austin Wright.

[Geraint]

Nested/sequential $refs should not be a problem for any implementation - if they are not working correctly in whatever library you're using, then do file a bug report.

tv4 should handle them fine.  If you decide to use that and have any trouble, I'm happy to help.

Geraint

P.S. - Eric: tv4 is for Node as well. :)

[Simon Heimler]

Hello Geraint,

ok, I've taken a look at tv4, looks like it has a lot of features I'm looking for.

But two things are missing (at least in the documentation?)
I'm not sure how to include tv4.async-jquery.js without getting an "Uncaught ReferenceError: tv4 is not defined" Error.

The other thing: Is it possible to seperate the asynchronous loading process from the validation process? This would make the library much more versataile for me. Since I'm just needing the complete schema first (to generate forms) I don't have an object to validate at this point in time.

(I see that you're speaking german too, if you like we could discuss this in private)

Greets,

Simon

--

[Eric G]

On Thursday, May 15, 2014 9:39:42 AM UTC-4, Simon Heimler wrote:

The other thing: Is it possible to seperate the asynchronous loading process from the validation process? This would make the library much more versataile for me. Since I'm just needing the complete schema first (to generate forms) I don't have an object to validate at this point in time.

That was one of my motivations for my implementation - which is split into a validator   and agent library, among other pieces - you can fetch/parse/dereference a schema on its own with the agent.  It does basically work, but like I said, is a bit of a pain and not tested by real use like tv4 is.

Eric

[Christopher J. White]

Simon, what you are describing below is precisely what $merge and $patch were designed for as proposed by Francis and discussed in a recent thread titled "additionalProperties and allOf, here we go again!"

I have implemented $merge in my JSON schema parser and find it serves the inheritance use case quite nicely. I haven't done $patch yet, which will support much more complicated variations, as I haven't had the need for more than what $merge provides.

...cj

--
You received this message because you are subscribed to the Google Groups "JSON Schema" group.
To unsubscribe from this group and stop receiving emails from it, send an email to json-schema...@googlegroups.com.

[Geraint]

On Thursday, 15 May 2014 14:39:42 UTC+1, Simon Heimler wrote:

Hello Geraint,

ok, I've taken a look at tv4, looks like it has a lot of features I'm looking for.

But two things are missing (at least in the documentation?)
I'm not sure how to include tv4.async-jquery.js without getting an "Uncaught ReferenceError: tv4 is not defined" Error.

The "async" file is a simple extension - you need to include the actual "tv4.js" as well.

 

 

The other thing: Is it possible to seperate the asynchronous loading process from the validation process? This would make the library much more versataile for me. Since I'm just needing the complete schema first (to generate forms) I don't have an object to validate at this point in time.

They are completely separate already.

"tv4.js" (the actual library) does all the validation, but does no fetching at all.  The "tv4.async-jquery.js" file is a demo browser-only extension to fetch schemas asynchronously on-demand.

If you want to do the asynchronous fetching up-front, that shouldn't be too complicated.  After you do "tv4.addSchema(url, schema);", then you can use "tv4.getMissingUris()" to get the URIs of all missing schemas.  Simple recursion should let you pre-fetch all relevant schemas.  I might have a go at writing a short script, but feel free to beat me to it - I expect it to only be 30 lines or so.

[Geraint]

On Thursday, 15 May 2014 16:38:09 UTC+1, Geraint wrote:

On Thursday, 15 May 2014 14:39:42 UTC+1, Simon Heimler wrote:

The other thing: Is it possible to seperate the asynchronous loading process from the validation process? This would make the library much more versataile for me. Since I'm just needing the complete schema first (to generate forms) I don't have an object to validate at this point in time.

If you want to do the asynchronous fetching up-front, that shouldn't be too complicated.  After you do "tv4.addSchema(url, schema);", then you can use "tv4.getMissingUris()" to get the URIs of all missing schemas.  Simple recursion should let you pre-fetch all relevant schemas.  I might have a go at writing a short script, but feel free to beat me to it - I expect it to only be 30 lines or so.

OK, here's something: https://gist.github.com/geraintluff/30f838e68ad1d3e4a591

It's completely untested, but it's a starting-point at least.  You should be able to do "tv4.asyncFetchSchemas(urls, callback)" and it'll get you the callback when all the schemas are loaded.

35 lines - I was close. :)

Geraint 

[Simon Heimler]

Hello Cristopher,

ok, thats interesting to know - i'll take a look into it!

My first (naive, but maybe intuitive) attempt how to implement Inheritance would look like this:

https://gist.github.com/Fannon/3307353eb2a00ccc01e2

For Merging the two objects together I would use jQuery.extend(), which is well known: http://api.jquery.com/jquery.extend/

If I replace $extends with $ref and use json-editor for schema-retrieval (which does use deep copy internally, i've just discovered), it already works like I want it to in simple usecases. 

What doesn't work: If I use $refs on my new attributes they do not load. $refs on the inherited attributes do. I guess this is a problem with the recursive algorithm.

Here is a demo online (the UI is german, sorry): 

The Class Circle: http://cbmodel.fannon.de/#model/Circle

Extends from: http://cbmodel.fannon.de/#model/_Shape

Greets,
Simon

--
You received this message because you are subscribed to a topic in the Google Groups "JSON Schema" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/json-schema/fM46FTFwR-U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to json-schema...@googlegroups.com.

[Simon Heimler]

Hello Geraint,

wow, thanks for the Gist! I'll try it out immediatly!

How do I access the completed, aggregated schema? 

I have no experience with this library and probably doing something wrong:

https://gist.github.com/Fannon/e718531f22f9791e1cfc

so long,

Simon

[Geraint]

On Thursday, 15 May 2014 17:27:19 UTC+1, Simon Heimler wrote:

Hello Geraint,

wow, thanks for the Gist! I'll try it out immediatly!

How do I access the completed, aggregated schema? 

I have no experience with this library and probably doing something wrong:

https://gist.github.com/Fannon/e718531f22f9791e1cfc

so long,

Simon

The schemas aren't "aggregated" - they're just all fetched.  tv4 resolves $refs at validation-time, so you don't need to aggregate them.

At least part of the issue is that you aren't extending properly.  Instead of using "$ref": ... in Circle.json (which says "replace this schema wholesale with another one"), you need to use "allOf": [{"$ref": ...}].

Anyway, once the schemas are loaded, then it just means that tv4 "knows about them".  So to validate an instance as a circle, you could just go:

tv4.validate(data, "/model/Circle.json");

and it should replace the URL with the Circle schema it fetched before.

[Simon Heimler]

Ah, you're right - using "allOf" eliminates now my last problem, too. Thanks for the tip!

[Sebastian Lasse]

regarding an editor - I am working on such a thing. 

However - loading external schemas requires node or how would you do it clientside ?

I plan to embed the source of popular schemas for clientside so that you can at least play around.

Please see my last post.