How to validate Jsonnet with Json Schema?

[Burak Emre Kabakcı]

We're writing Jsonnet via VSCode and using a plugin (https://github.com/heptio/vscode-jsonnet) to be able to integrate it into our workflow. It works great but there is one missing feature that prevents our workflow to be an ideal workflow everyone is dreaming about; Json schema integration.

VSCode already has integration with JSON Schema and it basically provides Java-like experience when writing JSON files via Intellisense. We can basically validate the output of Jsonnet in VScode but it's not really convenient because we need to backtrace the JSON file in order to find out the error line/position in Jsonnet manually and miss some important features such as type inference, smart suggestions, etc.

I wonder how hard this task is, do you need to have a source-map in Jsonnet in order to support JSON Schema or is there any "magic" way that we can use? I would be willing to integrate it into Vscode if we're able to validate Jsonnet via JSON Schema but this part is no different than a black box to me. :)

[Dave Cunningham]

Can you get the JSONschema validator to give error messages in terms of the structure into the JSON (i.e. the JSONpath)? This is a common need if you're generating JSON (even if you were using YAML with Jsonschema).  I have used JSONschema in Python and it does give you the path, not the line:col.
https://python-jsonschema.readthedocs.io/en/stable/errors/#jsonschema.exceptions.ValidationError.relative_path

--
You received this message because you are subscribed to the Google Groups "Jsonnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/f53440b4-9ed7-4b51-9a98-c2f744b5fcc9%40googlegroups.com.

[Burak Emre Kabakcı]

Yes, there is even a library that produces the line:col from the JSON Schema errors but the Jsonnet compiler only outputs JSONpath. See: https://github.com/atlassian/better-ajv-errors 

Would it be easy to add a flag into the Jsonnet compiler that validates and throws an appropriate exception using the JSON Schema input or is there any alternative solution?

[Dave Cunningham]

The problem when you're using actual programs to generate JSON is that a single line:col in the original program is not enough to help you debug invalid output.  The problem is harder than e.g. mapping ASM lines back to an original C program.  Even a full stack trace is not enough because it does not track the data dependencies (i.e. through variables / parameters).  So using JSONpath in errors is actually best because at least it gives you something that you can actually use to debug, no matter what the input.

What file line would you give for:

local func1(letter) = letter + 'ara'

local func2(letter) = letter + 'raph'

{

  content: {

    type: func1('t') + func2('g'),

  }

}

and what would you give for:

local func1(letter) = letter + 'ora'

local func2(letter) = letter + 'raph'

{

  content: {

    type: func1('p') + func2('g'),

  }

}

How would the system know the difference?  And this is a relatively easy case where stack traces would be sufficient. What about:

local word = 'plate'

local index = 10 - 3 * 2 - 3;
local func1(letter) = letter + 'ara'

local func2(letter) = letter + 'raph'

{

  content: {

    type: func1(word[index]) + func2('g'),

  }

}

You can see how this degenerates into the problem of "my 1 million line program produced 4 instead of 5, where is the bug?"

For me, an error message like "content.type was "laragraph" but expected "paragraph" is easy to implement and works well enough.  If it's hard to get from that to the right place in your Jsonnet file, then probably the Jsonnet is too complicated or not modular enough (the above example is definitely "too complicated"!).  If you're writing modular Jsonnet then test cases for those libraries help a lot to pick up the deep bugs and asserts can be used to check preconditions of functions / mixin fields.  There has been previous discussion about some self.validateJsonSchema(value, schema) that could be used in asserts but the problem there was compatible implementations of JSON schema in different languages.  Maybe that is no-longer an issue.

--

You received this message because you are subscribed to the Google Groups "Jsonnet" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/2cd0cb5b-75f4-405b-9321-1505e084f209%40googlegroups.com.

[Burak Emre Kabakcı]

Thanks for the detailed explanation, Dave. 

I believe that the validation should not actually be that sophisticated. What we can do is to produce the JSON output from a Jsonnet file, validate it using a JSON Schema and extract the JSONPath along with the error message. That way, we know the exact Jsonnet file and the JsonPath we need to highlight. The Jsonnet file might be dynamically generated via functions but that's OK. 

The compiler just need to try providing the line:col from a JsonPath and do a `best effort` here. It can try to process the Jsonnet file and return the closest parent field's line:col when it fails the find the children of a field given by Jsonnet. We can highlight that line:col in VSCode. Even if it fails, a generic error message for the file works but I think that the example that you have given `"content.type was "laragraph" but expected "paragraph"` covers most of the use-cases required to provide a seamless experience in an editor.

To unsubscribe from this group and stop receiving emails from it, send an email to jso...@googlegroups.com.

[Dave Cunningham]

Let me paraphrase: what is a best effort approach for doing a static analysis on a Jsonnet file that would identify the part of the file responsible for the longest possible prefix of the JSON path.  E.g.

$.foo.bar

in

1 local x = { bar: 10 }

2 

3 {

4   foo: x,
5 }

Might be 4:8 because the static analysis understands constant values but not variables so could only understand the $.foo prefix of the path.

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/9b1a581a-6ec5-449f-a90c-e6b75da362e4%40googlegroups.com.

[Burak Emre]

Correct, I believe that would be enough for VSCode.

[Dave Cunningham]

That shouldn't be too hard (from a programming languages perspective). We don't really have a framework for doing that kind of analysis yet, but it's not hard to just visit the ASTs and do something basic.  I would also only do it in the Go version though at this point.  All tooling should be done there because it's easier.  Ideally we would build up a framework there to allow more tooling to be written and improve the ecosystem.

[Angus Lees]

If I understand correctly: In a typical complex setup, this would just give you the first "import" point, which would almost always not be useful...

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/9b1a581a-6ec5-449f-a90c-e6b75da362e4%40googlegroups.com.

[Dave Cunningham]

This is OK:

local foo = import "...";

{
 ...
}

But something like this would fail pretty hard:

{

  foo: import "...",

  bar: import "...",

}

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/d8cb00bd-25a7-44f7-b9d9-85e062280eb0%40googlegroups.com.

[Burak Emre Kabakcı]

Take the following script called main.jsonnet as an example and assume that :

main.jsonnet:

{

  foo: import "foo.jsonnet"

}

foo.jsonnet:

{

  bar: 1

}

And our JSON Schema for main.jsonnet expects the value of `foo.bar` to be 2 and we're compiling/validating that file I would expect the IDE to highlight the error in `foo` property of main.jsonnet because I'm *compiling* that file, not foo.jsonnet. There's nothing wrong with foo.jsonnet because the JSON Schema is valid for only main.jsonnet, it's the file that needs to provide data which is valid for the JSON Schema. Otherwise, it would be confusing for me. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/d8cb00bd-25a7-44f7-b9d9-85e062280eb0%40googlegroups.com.

[Burak Emre Kabakcı]

Dave, what's your opinion on schema validation with Jsonnet? For the last few months, we have been evaluating the Jsonnet and its alternatives and due to its completeness and maturity, we continue to invest our time into Jsonnet. Our use-case is a bit different though, we allow users to model their data with Jsonnet and build up user analytics interfaces and the approach is rather similar to LookML from Looker. The only missing part for us is the schema validation which is not trivial. We also have looked into the Cuelang but it's not mature and I'm not sure types in Cue is the best way for our approach, it might be a bit over-engineering at least for our use-case.

Having said that, Json Schema is well known and feature-complete. It plays well with many IDEs and allows us to have advanced features such as conditional rules. (Our schema: https://json-schema-visualizer.netlify.com/) The most-used IDEs already makes use of most of the features like enums to be able to provide features such as code suggestions, go-to-references, suggestions in order to fix problems in JSON data etc and validate the data at the end.

Would you consider embedding Json Schema into Jsonnet or do you think that it's a better approach for Jsonnet to provide line:col from JsonPath and vice-versa so that one can make it work with Json Schema in IDEs like VSCode?

On Tuesday, August 27, 2019 at 9:18:51 PM UTC+3, Burak Emre Kabakcı wrote:

[Dave Cunningham]

It doesn't look like that's actually a link to your schema.

Both of those options are pretty hard to do.  I assume you want IDE support in both cases. The IDE cannot just execute the code, so dynamic approaches (like assertions) are out, even though (with better error messages) they might be sufficient for commandline use.  A static analysis or type system is needed.  They're basically the same thing at an abstract level - it's called semantic modelling and fortunately I did my PhD in this area! :)

JSON Schema would be very challenging to integrate into the language as a static type system.  It supports a lot of constraints (e.g. integer ranges, the conditional stuff you mentioned, regular expressions) that are considered very hard to implement in a static system.  You've got to worry about the performance (CPU and RAM use) of your type inference algorithm as well as if it's even technically possible (decidable).  It is trivial to verify them on an actual value, but not trivial to verify that an unexecuted expression will evaluate to a valid value.

However, a simpler type system is certainly within reach.  Something like https://github.com/google/pytype for Jsonnet would be a good fit.  It'll need structural types (possibly intersection types) and would have to handle the mixins, but all of that has been done before in other research projects (e.g. check out the language Forsythe and the original Mixins paper).  But then you won't get those complicated types that you seem to be using, so my question is could you make do with a simpler type system?

--

You received this message because you are subscribed to the Google Groups "Jsonnet" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jsonnet+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jsonnet/cfb7489e-3b59-4cd1-8737-588e6b831f54%40googlegroups.com.