Report on what of the schema gets used (and not used)

[pbeng...@mozilla.com]

As per the sample code in the documentation, https://python-jsonschema.readthedocs.io/en/latest/#module-jsonschema
it defines a schema for `/name` and `/price`.
Then, in the example it tests the schema with:

    validate({"name" : "Eggs", "price" : 34.99}, schema)

So both of those values get checked. But if I was to do:

    validate({"name" : "Eggs"}, schema)

Only the `/name` part of the schema gets "exercised".

I would like to inject or hack something (perhaps a subclass of Draft4Validator!) so that I can get insight into which paths were tested (at least once) and which paths were never tested.

Any ideas?

[Julian Berman]

Subclassing isn't a supported (or recommended) mechanism of extension :)

But what's your goal here? You can certainly use jsonschema.validators.extend and wrap all of a validator class's validator functions with some wrapper that keeps track of what is visited, or you can wrap your instance to be some instrumented mapping that keeps track of what is visited, or you can use Python's tracing features to keep track of what functions are being called, or ... etc :)

What will you do with the info basically?

--
You received this message because you are subscribed to the Google Groups "jsonschema - An implementation of JSON Schema for Python" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jsonschema+unsubscribe@googlegroups.com.
To post to this group, send email to jsons...@googlegroups.com.

[Peter Bengtsson]

The point is that we have a JSON Schema which needs to be validated by picking random objects from our database.

It could be that someone changes the JSON Schema and adds a new property but we're unlucky when we pick random objects that we fail to ever test certain parts of the JSON Schema.

We have a little python script in the repo where we maintain the .json file which is the JSON Schema. Developers are supposed to run this after having edited the .json file.

So, how do I use jsonschema.validators.extend in a way that I don't need to list each and every validator in code manually?

--

Peter Bengtsson
Mozilla Tools & Services

[Julian Berman]

I'm still not 100% sure I follow -- you have a schema, but you want to make sure that all your existing data passes it even if someone makes changes? Or you're trying to prevent people from forgetting to add properties to `required` in the schema?

The answer to your last question though is the .VALIDATORS attribute of any validator class.

[Peter Bengtsson]

On Sun, Mar 26, 2017 at 9:49 AM, Julian Berman <jul...@grayvines.com> wrote:

I'm still not 100% sure I follow -- you have a schema, but you want to make sure that all your existing data passes it even if someone makes changes? Or you're trying to prevent people from forgetting to add properties to `required` in the schema?

The use case is that I want to find out which parts of the JSON schema ever got "exercised" or "used".

For example, if someone extends the schema with `{"totall_memory_max": {"type": "integer", "description": "..."}}` and then tests a bunch of data they might completely miss the fact that they have a typo on "totall" when it's supposed to be "total". And perhaps the data had a piece of data like this: `{"total_memory_max": "notanumber"}` it would go unnoticed because of the typo.

If I can extend validators to list all the keys in the JSON Schema that were execised (and which were not) it might help people understand that they've added/modified something in the schema that was never seen.

 

The answer to your last question though is the .VALIDATORS attribute of any validator class.

Do you have an example of that?
 

--

Peter Bengtsson
Mozilla Engineering Operations

[Julian Berman]

Ah. I see. I think the actual solution to that sort of problem probably revolves around writing unit tests, but I could forsee something measuring schema coverage as being useful.

http://python-jsonschema.readthedocs.io/en/latest/validate/#jsonschema.IValidator.VALIDATORS is the docs on the attribute I mentioned, the values of that dict are the functions that are called for each validator.

You can wrap each of those functions with some tracing to keep track of what you've seen, which is probably a decent start there.

Would be interested to see what you come up with.