"There was an error while encoding violations"

[fcb...@gmail.com]

Hi, I'm getting these exceptions while trying to do an ICV.  My database has a couple million triples so it will be hard to reduce this to a small sample to reproduce the problem.  I'm trying to understand what the error means, though.  What type of encoding is it talking about? Is it some internal encoding or are we talking about character encodings, or something else?

Thanks,

Fabricio

com.complexible.stardog.cli.CliException: There was an error while encoding violations
    at com.complexible.stardog.cli.impl.ConnectionCommand.call(ConnectionCommand.java:118)
    at com.complexible.stardog.cli.CLIBase.execute(CLIBase.java:54)
    at com.complexible.stardog.cli.CLI.main(CLI.java:99)
Caused by: com.complexible.stardog.StardogException: There was an error while encoding violations
    at com.complexible.stardog.protocols.snarl.icv.client.ICVClientImpl.executeICV(ICVClientImpl.java:161)
    at com.complexible.stardog.protocols.snarl.icv.client.ICVClientImpl.getViolationBindings(ICVClientImpl.java:127)
    at com.complexible.stardog.protocols.snarl.icv.client.SNARLICVConnection.getViolationBindings(SNARLICVConnection.java:108)
    at com.complexible.stardog.icv.cli.ICVValidate.execute(ICVValidate.java:72)
    at com.complexible.stardog.cli.impl.ConnectionCommand.call(ConnectionCommand.java:106)
    ... 2 more
Caused by: com.complexible.common.protocols.client.ClientException:
There was an error while encoding violations
    at com.complexible.common.protocols.client.rpc.DefaultRPCClient.get(DefaultRPCClient.java:344)
    at com.complexible.common.protocols.client.rpc.DefaultRPCClient.execute(DefaultRPCClient.java:321)
    at com.complexible.stardog.protocols.snarl.client.DelegatingSNARLClient.execute(DelegatingSNARLClient.java:235)
    at com.complexible.stardog.protocols.snarl.icv.client.ICVClientImpl.executeICV(ICVClientImpl.java:156)
    ... 6 more
Caused by: com.complexible.common.protocols.EncodingException: There was an error while encoding violations
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at com.complexible.barc.DefaultThrowableCodec.decode(DefaultThrowableCodec.java:89)
    at com.complexible.barc.BigPacketCodec.toException(BigPacketCodec.java:393)
    at com.complexible.barc.BigPacketCodec.decode(BigPacketCodec.java:360)
    at com.complexible.barc.BigPacketDecoder.channelRead(BigPacketDecoder.java:75)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:182)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    at java.lang.Thread.run(Thread.java:745)

[Michael Grove]

On Fri, Aug 21, 2015 at 3:05 PM, <fcb...@gmail.com> wrote:

Hi, I'm getting these exceptions while trying to do an ICV.  My database has a couple million triples so it will be hard to reduce this to a small sample to reproduce the problem.  I'm trying to understand what the error means, though.  What type of encoding is it talking about? Is it some internal encoding or are we talking about character encodings, or something else?

That indicates there was an error encoding them for transmission from the server back to the client.  There should be a corresponding stack trace in the server log which will pinpoint where the error occurred.  You you please provide that trace?

Thanks.

Mike

 

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Fabricio]

Is it the one pointed to by LOG_FILE? It so, there doesn't seem to be
anything relevant there.

Cheers,
Fabricio

[Michael Grove]

On Fri, Aug 21, 2015 at 4:34 PM, Fabricio <fcb...@gmail.com> wrote:

Is it the one pointed to by LOG_FILE?  It so, there doesn't seem to be
anything relevant there.

Are your client and server running the same version of Stardog?

Cheers,

Mike

[Fabricio]

Yep, I just downloaded it and ran both locally on my machine.

[Fabricio]

Here's how to reproduce it:

~/bin/stardog-3.1.4/bin/stardog-admin server start
~/bin/stardog-3.1.4/bin/stardog-admin db drop wn
~/bin/stardog-3.1.4/bin/stardog-admin db create -n wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog data add wn wordnet-en.nt
~/bin/stardog-3.1.4/bin/stardog data add wn own-pt.nt
~/bin/stardog-3.1.4/bin/stardog data add wn skos.rdf
~/bin/stardog-3.1.4/bin/stardog-admin icv add -v wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog icv validate -v -r wn <--- this is
there the exception is thrown.

You can download wn30.ttl, wordnet-en.nt and own-pt.nt from
https://github.com/own-pt/openWordnet-PT

You can download SKOS from http://www.w3.org/TR/skos-reference/skos.rdf

Cheers,
Fabricio

[Michael Grove]

On Fri, Aug 21, 2015 at 7:05 PM, Fabricio <fcb...@gmail.com> wrote:

Here's how to reproduce it:

~/bin/stardog-3.1.4/bin/stardog-admin server start
~/bin/stardog-3.1.4/bin/stardog-admin db drop wn
~/bin/stardog-3.1.4/bin/stardog-admin db create -n wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog data add wn wordnet-en.nt
~/bin/stardog-3.1.4/bin/stardog data add wn own-pt.nt
~/bin/stardog-3.1.4/bin/stardog data add wn skos.rdf
~/bin/stardog-3.1.4/bin/stardog-admin icv add -v wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog icv validate -v -r wn  <--- this is
there the exception is thrown.

You can download wn30.ttl, wordnet-en.nt and own-pt.nt from
https://github.com/own-pt/openWordnet-PT

You can download SKOS from http://www.w3.org/TR/skos-reference/skos.rdf

I think I've tracked down what's going on here, and with what Alexandre reported.  You really probably don't mean for wn30.ttl to be used as contraints, many of the axioms in that file do not make sense as constraints.  Some of them we'll filter out automatically, such as ones using annotations (ticket #2455), but you're best off directly writing the constraints you want to validate.

What's happening is that there are so many constraint violations, well over 200, that Stardog is running out of memory computing all of the violations and sending them across the wire.  This seems to be the encoding problem you're seeing, it's OOMing while writing the result, and what appears to be "stuck" validation as reported by Alexandre; icv validate tries to print all of the violations.

We will improve the streaming of results back to the client to avoid the memory issue in a future release (ticket #2456), but in the meantime, you should try authoring just the constraints you mean to validate.

[Alexandre Rademaker]

Hi Michael,

On Aug 24, 2015, at 11:54 AM, Michael Grove <mi...@complexible.com> wrote:

On Fri, Aug 21, 2015 at 7:05 PM, Fabricio <fcb...@gmail.com> wrote:

Here's how to reproduce it:

~/bin/stardog-3.1.4/bin/stardog-admin server start
~/bin/stardog-3.1.4/bin/stardog-admin db drop wn
~/bin/stardog-3.1.4/bin/stardog-admin db create -n wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog data add wn wordnet-en.nt
~/bin/stardog-3.1.4/bin/stardog data add wn own-pt.nt
~/bin/stardog-3.1.4/bin/stardog data add wn skos.rdf
~/bin/stardog-3.1.4/bin/stardog-admin icv add -v wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog icv validate -v -r wn  <--- this is
there the exception is thrown.

You can download wn30.ttl, wordnet-en.nt and own-pt.nt from
https://github.com/own-pt/openWordnet-PT

You can download SKOS from http://www.w3.org/TR/skos-reference/skos.rdf

I think I've tracked down what's going on here, and with what Alexandre reported.  You really probably don't mean for wn30.ttl to be used as contraints, many of the axioms in that file do not make sense as constraints. 

Some of them we'll filter out automatically, such as ones using annotations (ticket #2455), but you're best off directly writing the constraints you want to validate.

Well, but it would be very laborious  to separate the constraints from the model definition.

What's happening is that there are so many constraint violations, well over 200, that Stardog is running out of memory computing all of the violations and sending them across the wire.  This seems to be the encoding problem you're seeing, it's OOMing while writing the result, and what appears to be "stuck" validation as reported by Alexandre; icv validate tries to print all of the violations.

Can we give to the tool the upper limit for the number of violations to be returned? In this way, we don't need to create different files for each constraint but control the amount of "errors" to receive back.

We will improve the streaming of results back to the client to avoid the memory issue in a future release (ticket #2456), but in the meantime, you should try authoring just the constraints you mean to validate.

We don't know in advance the constraints that we want to validate, we are using the tool precisely to help us to identify which constraints were not respected in the data.

Best,

Alexandre

[Michael Grove]

On Mon, Sep 7, 2015 at 6:02 PM, Alexandre Rademaker <arade...@gmail.com> wrote:

Hi Michael,

On Aug 24, 2015, at 11:54 AM, Michael Grove <mi...@complexible.com> wrote:

On Fri, Aug 21, 2015 at 7:05 PM, Fabricio <fcb...@gmail.com> wrote:

Here's how to reproduce it:

~/bin/stardog-3.1.4/bin/stardog-admin server start
~/bin/stardog-3.1.4/bin/stardog-admin db drop wn
~/bin/stardog-3.1.4/bin/stardog-admin db create -n wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog data add wn wordnet-en.nt
~/bin/stardog-3.1.4/bin/stardog data add wn own-pt.nt
~/bin/stardog-3.1.4/bin/stardog data add wn skos.rdf
~/bin/stardog-3.1.4/bin/stardog-admin icv add -v wn wn30.ttl
~/bin/stardog-3.1.4/bin/stardog icv validate -v -r wn  <--- this is
there the exception is thrown.

You can download wn30.ttl, wordnet-en.nt and own-pt.nt from
https://github.com/own-pt/openWordnet-PT

You can download SKOS from http://www.w3.org/TR/skos-reference/skos.rdf

I think I've tracked down what's going on here, and with what Alexandre reported.  You really probably don't mean for wn30.ttl to be used as contraints, many of the axioms in that file do not make sense as constraints. 

Some of them we'll filter out automatically, such as ones using annotations (ticket #2455), but you're best off directly writing the constraints you want to validate.

Well, but it would be very laborious  to separate the constraints from the model definition.

Perhaps, but it's usually the case that there are *many* axioms in the ontology that are not desirable, or even suitable, as constraints.  Further, you can be more expressive with constraints than you can be with your ontology.  And the modeling is different; closed world vs. open world.

 

What's happening is that there are so many constraint violations, well over 200, that Stardog is running out of memory computing all of the violations and sending them across the wire.  This seems to be the encoding problem you're seeing, it's OOMing while writing the result, and what appears to be "stuck" validation as reported by Alexandre; icv validate tries to print all of the violations.

Can we give to the tool the upper limit for the number of violations to be returned? In this way, we don't need to create different files for each constraint but control the amount of "errors" to receive back.

Not currently.

 

We will improve the streaming of results back to the client to avoid the memory issue in a future release (ticket #2456), but in the meantime, you should try authoring just the constraints you mean to validate.

We don't know in advance the constraints that we want to validate, we are using the tool precisely to help us to identify which constraints were not respected in the data.

Then you might consider creating the constraints one by one and using the Java API to validate the data against each constraint rather than loading them into the database and checking them all at once.  This is probably a more workable approach given the number of constraints and how generally invalid the data seems to be wrt to those constraints.

Cheers,

Mike

 

Best,

Alexandre

[Evren Sirin]

Also note that an axiom might be suitable as a constraint but if you
also add it to your ontology for reasoning they will cancel each other
out. For example, suppose you have a constraint

<p> rdfs:domain <A>

and validate your data that has a single triple

<a> <p> <b>

and you'll get a violation saying <a> is not of type <A>. Now if you
add the domain axiom to your database, the violation will go away
because then reasoning will kick in and infer that <a> is of type <A>.
That's why having some axioms both as a constraint and in your
ontology doesn't make sense.

Best,
Evren

[Alexandre Rademaker]

Hi Evren and Michael,

Thanks for clarifying. Indeed, now it is much clearer to me the argument in favor of having the constraints separated from the 'ontology'. The OWA and CWA semantics are per axiom, right?

I have already understood that all axioms in the file provided in the 'db creation', were defined with OWA semantics. But if the same axioms are provided as constraints lately, the OWA version is not overwritten by the CWA?

What are the minimum information that we must provide in the file passed to 'db creation'? Classes and properties declarations only?

Best,
Alexandre

[Michael Grove]

On Wed, Sep 9, 2015 at 7:46 PM, Alexandre Rademaker <arade...@gmail.com> wrote:

Hi Evren and Michael,

Thanks for clarifying. Indeed, now it is much clearer to me the argument in favor of having the constraints separated from the 'ontology'. The OWA and CWA semantics are per axiom, right?

Right, all of the axioms in your ontology will be interpreted w/ OWA and your constraints, CWA.

 

I have already understood that all axioms in the file provided in the 'db creation', were defined with OWA semantics. But if the same axioms are provided as constraints lately, the OWA version is not overwritten by the CWA?

Correct.  The ontology is stored separate from any constraints.

 

What are the minimum information that we must provide in the file passed to 'db creation'? Classes and properties declarations only?

Minimum information?  An ontology is not required.  You can provide one when you create the database, or you can add it later.  You only need it if you wish to perform reasoning over your data.  

It's perfectly reasonable to have data and your ontology, or data and just constraints, or all three.  Whatever works best for your use case.

Cheers,

Mike

[Alexandre Rademaker]

Hi Michael and Evren,

I have just tried Stardog one more time with a simpler turtle file as constraints (see below). I created the database with no TBOX and further I added the file below as constraints. Unfortunately, even in this case, I had the same behaviour, Stardog hung and didn't produce any output or message in the log. This result is really disappointing since I was willing to mention Startdog in a paper we are working on.

The commands were

stardog-admin server start

stardog-admin db drop wn

stardog-admin db create -n wn

stardog data add wn wordnet-en.nt.gz

stardog data add wn own-pt.nt.gz

stardog-admin icv add -v wn wn30-simple.ttl

stardog icv validate -v -r wn

where wordnet.en.nt.gz and own-pt.nt.gz can be downloaded from https://github.com/own-pt/openWordnet-PT and the wn30-simple.ttl is:

---

@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .

@prefix owl: <http://www.w3.org/2002/07/owl#> .

@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

@prefix wn30: <https://w3id.org/own-pt/wn30/schema/> .

@prefix nomlex: <https://w3id.org/own-pt/nomlex/schema/> .

@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

xsd:integer

    a rdfs:Datatype .

xsd:nonNegativeInteger

    a rdfs:Datatype .

xsd:positiveInteger

    a rdfs:Datatype .

owl:Thing

    a owl:Class .

wn30:WordSense

    a rdfs:Class, owl:Class ;

    rdfs:subClassOf [

        a owl:Restriction ;

owl:onProperty wn30:inSynset ;

        owl:qualifiedCardinality "1"^^xsd:nonNegativeInteger ;

        owl:onClass wn30:Synset

    ], [

        a owl:Restriction ;

owl:onProperty wn30:word;

owl:qualifiedCardinality "1"^^xsd:nonNegativeInteger ;

owl:onClass wn30:Word

    ] .

wn30:word

    a rdf:Property, owl:ObjectProperty ;

    rdfs:domain wn30:WordSense ;

    rdfs:range wn30:Word ;

    owl:inverseOf wn30:sense .

wn30:containsWordSense

    a rdf:Property, owl:ObjectProperty ;

    rdfs:domain wn30:Synset ;

    rdfs:range wn30:WordSense ;

    owl:inverseOf wn30:inSynset .

wn30:Word

    a rdfs:Class, owl:Class ;

    rdfs:subClassOf [

        a owl:Restriction ;

        owl:onProperty wn30:sense ;

        owl:someValuesFrom wn30:WordSense

    ], [

        a owl:Restriction ;

owl:onProperty wn30:lexicalForm ;

        owl:qualifiedCardinality "1"^^xsd:nonNegativeInteger ;

        owl:onDataRange rdfs:Literal

    ] ;

    owl:disjointWith wn30:WordSense .

wn30:Synset

    a rdfs:Class, owl:Class ;

    rdfs:subClassOf owl:Thing, [

        a owl:Restriction ;

        owl:onProperty wn30:containsWordSense ;

        owl:someValuesFrom wn30:WordSense

    ] ;

    owl:disjointWith wn30:Word, wn30:WordSense .

---

Best,

Alexandre