SHACL validation report from ModelConstraintValidator example - Questions

[Jack Hodges]

I have looked at and run the ValidationExample from the github archive and have looked carefully at the SHACL archive ModelConstraintValidator. These are very helpful and I encourage anyone playing with SHACL to look at them. This said, I have run into a bit of a problem trying to extend the example. Here is a code fragment from a modified version of the Validation Example in the archive:

// Load the main data model

Model dataModel = JenaUtil.createMemoryModel();

dataModel.read(EDDLValidation.class.getResourceAsStream("/sh/project/vocab/Data.ttl"), "urn:dummy", FileUtils.langTurtle);

Model schemaModel = JenaUtil.createMemoryModel();

schemaModel.read(ValidationExample.class.getResourceAsStream("/sh/project/schema/Schema.ttl"), "urn:dummy", FileUtils.langTurtle);

// Load the shapes Model (here, includes the dataModel because that has shape definitions too)

Model shaclModel = SHACLSystemModel.getSHACLModel();

MultiUnion unionGraph = new MultiUnion(new Graph[] {

shaclModel.getGraph(),

schemaModel.getGraph()

});

Model shapesModel = ModelFactory.createModelForGraph(unionGraph);

// Make sure all sh:Functions are registered

SHACLFunctions.registerFunctions(shapesModel);

// Create Dataset that contains both the main query model and the shapes model

// (here, using a temporary URI for the shapes graph)

URI shapesGraphURI = URI.create("uri_path_to_schema_graph");

Dataset dataset = ARQFactory.get().getDataset(dataModel);

dataset.addNamedModel(shapesGraphURI.toString(), shapesModel);

So the questions here are as follows:

(1) does the schemaModel graph need to be a combined graph that includes any imports (such as separate SPIN graphs) within it or is it ok to be just the graph that has SHACL validation functions defined within it? I suspect that it must include the SPIN graph, and would that be done with a jena export from TBC of the related graphs?

(2) in the ValidationExample a MultiUnion graph was used because there were some SHACL definitions in the dataset. This would generally not be the case so this extra layer wouldn't be necessary, right?

(3) In my current case I am getting a result: @base <http://example.org/random> .

Is this the kind of result to expect if there are no constraint violations or have I done something wrong in the code fragment?

Thanks to anyone for any pointers.

Jack Hodges

[Holger Knublauch]

On 9/08/2016 9:31, Jack Hodges wrote:

I have looked at and run the ValidationExample from the github archive and have looked carefully at the SHACL archive ModelConstraintValidator. These are very helpful and I encourage anyone playing with SHACL to look at them. This said, I have run into a bit of a problem trying to extend the example. Here is a code fragment from a modified version of the Validation Example in the archive:

// Load the main data model

Model dataModel = JenaUtil.createMemoryModel();

dataModel.read(EDDLValidation.class.getResourceAsStream("/sh/project/vocab/Data.ttl"), "urn:dummy", FileUtils.langTurtle);

Model schemaModel = JenaUtil.createMemoryModel();

schemaModel.read(ValidationExample.class.getResourceAsStream("/sh/project/schema/Schema.ttl"), "urn:dummy", FileUtils.langTurtle);

// Load the shapes Model (here, includes the dataModel because that has shape definitions too)

Model shaclModel = SHACLSystemModel.getSHACLModel();

MultiUnion unionGraph = new MultiUnion(new Graph[] {

shaclModel.getGraph(),

schemaModel.getGraph()

});

Model shapesModel = ModelFactory.createModelForGraph(unionGraph);

// Make sure all sh:Functions are registered

SHACLFunctions.registerFunctions(shapesModel);

// Create Dataset that contains both the main query model and the shapes model

// (here, using a temporary URI for the shapes graph)

URI shapesGraphURI = URI.create("uri_path_to_schema_graph");

Dataset dataset = ARQFactory.get().getDataset(dataModel);

dataset.addNamedModel(shapesGraphURI.toString(), shapesModel);

So the questions here are as follows:

(1) does the schemaModel graph need to be a combined graph that includes any imports (such as separate SPIN graphs) within it or is it ok to be just the graph that has SHACL validation functions defined within it? I suspect that it must include the SPIN graph, and would that be done with a jena export from TBC of the related graphs?

To get this out of the way, the *shapes* graph needs to include at least:
- the W3C SHACL graph itself
- the DASH namespace

The latter includes the SPARQL queries and other info needed at runtime, while the former should be there anyway, to get the class hierarchy correct at runtime. I would assume the RDF systems graph should be included, too. All this is handled by SHACLSystemModel.getSHACLModel as a convenience. In addition to these, you certainly want to add your own shapes to that graph, i.e. the schema model.

I see no relation to SPIN here - so any SPIN imports should be optional when running SHACL. But yes, the simplest thing to do would be to "flatten" all graphs imported by TBC e.g. using the Export/Merge wizard.

(2) in the ValidationExample a MultiUnion graph was used because there were some SHACL definitions in the dataset. This would generally not be the case so this extra layer wouldn't be necessary, right?

The union graph is the combination of the general SHACL system graphs plus your actual data. The whole separation between the two graphs is only needed if you have your shapes (i.e. classes or schema) separated from instance data. By keeping them separate, the theory is that engines need to do less work because they don't really need the specific instances to figure out which shape declarations exist. If you are just playing with SHACL in a normal set up, I don't think this performance difference would be significant. So you can basically treat shapesModel = dataModel.

(3) In my current case I am getting a result: @base <http://example.org/random> .

Is this the kind of result to expect if there are no constraint violations or have I done something wrong in the code fragment?

An empty results graph would mean no violations were found. It's hard to guess whether this is correct without seeing the actual data and shapes.

HTH
Holger

[GMAIL]

We separate our schemes and vocabularies normally. Thank you for the clarifications Holger!

Sent from my Siemens iPhone

--
You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.
To post to this group, send email to topbrai...@googlegroups.com
---
You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to topbraid-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jack Hodges]

I need SPIN because my SHACL sh:constraint sh:sparql SPARQL query calls SPIN functions (some stock and some custom).

I flattened as suggested into a single turtle file.

I had some test values in the schema so I piped the schema back in as the test data and got some results (yay!).

Now the issue is that the results don't match what I am seeing in the TBC SHACL Validation tab. In TBC the SHACL Validation tab shows: Shape, Constraint, Message, Focus Node, Subject, Predicate, and Object. The Shape data is similar to the result in the test case, but none of the other SHACL property values are the same, most notably the Message and Focus Node values. In TBC the Focus Node (which seems to be the same as sh:subject) is bound to instances, and the Message (sh:message) value is bound to the message I defined and bound to the values my SPARQL query returns. And example of the results I am seeing from the test case is:

[ a                             sh:ValidationResult ;

  sh:focusNode                  eddl:BlockB ;

  sh:message                    "Value does not have class sh:PropertyConstraint" ;

  sh:path                       sh:property ;

  sh:severity                   sh:Violation ;

  sh:sourceConstraint           _:b0 ;

  sh:sourceConstraintComponent  sh:ClassConstraintComponent ;

  sh:sourceShape                sh:Shape ;

  sh:value                      [] 

] .

So now I am thinking that maybe I am using the wrong validator. Is it possible that I should be using the ResourceConstraintValidator instead of the ModelConstraintValidator? And if so, it takes a resource as the first argument unlike TBC which traverses the current graph.

Jack

To post to this group, send email to topbraid-users@googlegroups.com

---
You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.

[Holger Knublauch]

On 10/08/2016 2:51, Jack Hodges wrote:
> I need SPIN because my SHACL sh:constraint sh:sparql SPARQL query
> calls SPIN functions (some stock and some custom).

The SPIN functions can be registered with the SPARQL engine separately
and do not need to be in the same graph as the SHACL data. The two
worlds interact only via the ARQ SPARQL engine's functions registry.

So are you saying the result instances are different, i.e. they report
different violations? Or is just the format (e.g. sh:focusNode vs.
sh:subject) different? The latter is easy to explain: the SHACL API is
ahead of the TBC release cycle, and only the API follows the very latest
syntax changes to SHACL.

Holger

[Jack Hodges]

That is what I am saying. The validation example is reporting violations on classes (focus nodes are all classes) while TBC is reporting violations on instances. I am trying to reproduce the results that TBC shows.

Jack

Sent from my iPad

> --
> You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.

> To post to this group, send email to topbrai...@googlegroups.com

> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to topbraid-user...@googlegroups.com.

[Holger Knublauch]

On 10/08/2016 12:53, Jack Hodges wrote:
> That is what I am saying. The validation example is reporting violations on classes (focus nodes are all classes) while TBC is reporting violations on instances. I am trying to reproduce the results that TBC shows.

TBC has a green and a blue button in the SHACL Validation view. The
green will validate everything including the shapes and classes. The
blue only the instances. In the API this corresponds roughly to the
validateShapes boolean flag.

I cannot tell more since I don't know your shapes file, and which
violations you expected.

Holger

[Jack Hodges]

Right, I forgot about that. I always hit the blue button. I'll try setting validateShapes to false. Thank you!

Jack

Sent from my iPad

[Jack Hodges]

I removed the SPIN content from my validation.

I removed the instances from my graphs to a new graph.

I changed the validateShapes flag to false.

Locally in TBC I am still seeing different response (albeit a bit different now that validateShapes is set to false. For one file I get no response at all, but that file, when run in TBC, produces 64K results. The other file, which produces about 200 results in TBC, produces 1 in the project. Attached is a screen shot of the TBC results for the latter file.

Looking at these results they appear very different, for several reasons. The most notable now is that only the sh: validations are being run in my jena case. Perhaps this is related to the fact that I can no longer run SPINModuleRegistry.get().init(); because connection is refused. Since many of my SHACL templates are defined in my SPIN graph, if I cannot get them registered they may not be available.

Here is my initializeSPIN method, glomed together from various examples, which is called first in my main for the ModelConstraintValidator:

    /**

     * initializeSPIN: Get all of the related SPIN modules and our SPIN stuff and load

     *                 them into the SPINModuleRegistry

     */

    private static void initializeSPIN() {

        SPINModuleRegistry.get().init();

        // Create an empty OntModel importing SP

        System.out.println("Loading EDDL SPIN ontology...");

        Model eddlSPINModel = JenaUtil.createMemoryModel();

        eddlSPINModel.read(ValidationExample.class.getResourceAsStream("/sh/eddl/spin/SPIN_SSF_eddl-v1.0.spin.ttl"), null, FileUtils.langTurtle);

        // Load OWL RL library from the web

        System.out.println("Loading OWL RL ontology...");

        OntModel owlrlModel = loadModelWithImports("http://topbraid.org/spin/owlrl-all");

        // Register any new functions defined in OWL RL

        SPINModuleRegistry.get().registerAll(owlrlModel, null);

        // Build one big union Model of everything

        System.out.println("Creating MultiUnion model from eddlSPIN, owlrlModel, SPL, SPIN, and SP...");

        MultiUnion multiUnion = JenaUtil.createMultiUnion(new Graph[] {

            eddlSPINModel.getGraph(),

            owlrlModel.getGraph(),

SPL.getModel().getGraph(),

SPIN.getModel().getGraph(),

SP.getModel().getGraph()

});

        System.out.println("Assigning into spinModel the MultiUnion model...");

spinModel = ModelFactory.createModelForGraph(multiUnion);

// Register locally defined functions (none exist, but may in the future)

        System.out.println("Registering everything in spinModel into the SPIN Module Registry...");

SPINModuleRegistry.get().registerAll(spinModel, null);

return;

>>> To post to this group, send email to topbraid-users@googlegroups.com

>>> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.

>>> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.

>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.

> To post to this group, send email to topbraid-users@googlegroups.com

> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.

[Jack Hodges]

The SPINModuleRegistry.get().init() problem was a Siemens security issue which I got rid of but it didn't change the result I posted, which suggests that the SPIN functions are not available in the SPARQL queries in my SHACL templates.

>>> To post to this group, send email to topbrai...@googlegroups.com

>>> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.

> To post to this group, send email to topbrai...@googlegroups.com

> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.
To post to this group, send email to topbraid-users@googlegroups.com
---
You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jack

[Holger Knublauch]

On 16/08/2016 10:07, Jack Hodges wrote:

I removed the SPIN content from my validation.

I removed the instances from my graphs to a new graph.

I changed the validateShapes flag to false.

Locally in TBC I am still seeing different response (albeit a bit different now that validateShapes is set to false. For one file I get no response at all, but that file, when run in TBC, produces 64K results. The other file, which produces about 200 results in TBC, produces 1 in the project. Attached is a screen shot of the TBC results for the latter file.

TBC and the SHACL API are out of synch right now. Syntax has changed etc. This makes comparing results quite hard right now. I also don't have a complete picture of what you are doing. When running SHACL in TBC, are your SPIN functions registered globally (e.g. stored in .spin.ttl) files so that they are known? Could you try to make some test queries using the SPARQL view, with only the SHACL file open in TBC?

Holger

To post to this group, send email to topbrai...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.

[Jack Hodges]

Yes, all of my SPIN functions are in .spin.ttl files, and they all work just fine in TBC (I always test them in SPARQL view before making them SPIN functions, and then again after I make them SPIN functions, and I test them in SHACL in TBC as well).

I recognize and remember your warnings about the synchronization problem, but I think that the problem I am experiencing is independent of that problem. I can share the projects with you offline if that would help.

Jack

>>> To post to this group, send email to topbrai...@googlegroups.com
>>> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
>>> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.
> To post to this group, send email to topbrai...@googlegroups.com

> --- You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.
To post to this group, send email to topbraid-users@googlegroups.com
---

You received this message because you are subscribed to the Google Groups "TopBraid Suite Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Group "TopBraid Suite Users", the topics of which include the TopBraid Suite family of products and its base technologies such as SPARQLMotion, SPARQL Web Pages and SPIN.
To post to this group, send email to topbraid-users@googlegroups.com
---
You received this message because you are subscribed to a topic in the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/topbraid-users/4suqs8RewM0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to topbraid-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jack