Stardog VM Size and RAM for VirtualGraphs with 50 million records

4 views
Skip to first unread message

charbe...@gmail.com

unread,
Sep 20, 2016, 6:08:41 PM9/20/16
to Stardog
Hello,

I am trying to figure out what is the best configuration for Stardog in order to query a virtual graph which is pretty simple with only 3 columns.
The virtual graph is on a Azure SQL Server P1 configuration.

I installed Stardog on a linux VM on Azure with DS2 v2 with 2 cores and 7 GB memory. I am having connection timeouts on the virtual graph part even after configuring the timeout to 30 000 sec.


my qyeru looks like the following,

SELECT DISTINCT ?a 
WHERE 
{
  ?a a qt:Server.
  ?a qt:hasPoint ?point.
  ?point qt:hasType pdm:WHr_I.


    GRAPH <virtual://db2> 
    {
     ?b qt:hasValue ?value.    
    }
 filter (?a = ?b)          
}
group by ?a

Thank you,
Charbel

Jess Balint

unread,
Sep 20, 2016, 6:44:54 PM9/20/16
to sta...@clarkparsia.com
Charbel,

How many unique results do anticipate will be returned from the virtual graph query? The filter will be executed in Stardog. We don't yet have an optimization to handle this type of join between the local RDF graph and a remote RDBMS viewed as a virtual graph.

Jess

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+unsubscribe@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

Charbel Kaed

unread,
Sep 20, 2016, 6:52:55 PM9/20/16
to Stardog
Hi Jess,

I am expecting between 500.000 to 1 million records.

would you recommend that I move the data (50 million record) to Stardog instead?

Thank you,

Jess Balint

unread,
Sep 20, 2016, 6:58:07 PM9/20/16
to sta...@clarkparsia.com
Charbel,

You have several options. You can materialize the entire virtual graph in Stardog or only the "qt:hasValue" property. The queries will certainly be faster if the data is in Stardog. The balance is between query speed and copying of data to Stardog. If your dataset is (relatively) static, materializing in Stardog is a good option. How many unique values for "?a" do you anticipate?

Jess

Charbel Kaed

unread,
Sep 20, 2016, 7:08:34 PM9/20/16
to Stardog
Jess,

For now, I have 230 unique values for a. But 50 million records in the virtual graph.
I can consider the data is static for now. However, I received 230 entries every 5 minutes.

Jess Balint

unread,
Sep 20, 2016, 7:27:56 PM9/20/16
to sta...@clarkparsia.com
Since your virtual graph data is static, it would be good to import into Stardog. The virtual graph data will be available to join with new "qt:Server" individuals as they arrive. You can materialize a subset of virtual graph data in Stardog using SPARQL (e.g. insert { graph <localCopy> { ?b qt:hasValue ?v } } where { graph <virtual://db2> { ?b qt:hasValue ?value } }) or the entire virtual graph with the standard ADD/COPY commands as detailed in [1].

Jess

Zachary Whitley

unread,
Sep 20, 2016, 7:40:43 PM9/20/16
to sta...@clarkparsia.com
Just a couple of things to add. Do you have any indexes on the table? Maybe I'm missing something but wouldn't the following be equivalent but possibly faster?

SELECT DISTINCT ?a 
WHERE 
{
  ?a a qt:Server.
  ?a qt:hasPoint ?point.
  ?point qt:hasType pdm:WHr_I.


    GRAPH <virtual://db2> 
    {
     ?a qt:hasValue ?value.    
    } 
}
group by ?a

---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+u...@clarkparsia.com.

Jess Balint

unread,
Sep 20, 2016, 7:48:17 PM9/20/16
to sta...@clarkparsia.com
On Tue, Sep 20, 2016 at 6:40 PM, Zachary Whitley <zachary...@gmail.com> wrote:
Just a couple of things to add. Do you have any indexes on the table? Maybe I'm missing something but wouldn't the following be equivalent but possibly faster?


Stardog will execute both forms of the query in the same way. The issue is that virtual graph queries joined with the local RDF graph aren't well optimized. The more selective the virtual graph query the better.

Jess

---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.

charbe...@gmail.com

unread,
Sep 20, 2016, 8:26:28 PM9/20/16
to sta...@clarkparsia.com
 Thank you Jess and Zachary,

@Zachary: I will try this query to check if it makes any difference.

@Jess: I would prefer not to import the virtual graph into Stardog as it is getting updated every 5 min. But will try it to have an idea about performance.
I guess with the materialization operation copies the data from the RDBMS and indexes it locally in Stardog?

Thank you,
Charbel



Charbel

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "Stardog" group.
To unsubscribe from this topic, visit https://groups.google.com/a/clarkparsia.com/d/topic/stardog/J00Fc89Q5aw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stardog+u...@clarkparsia.com.

Jess Balint

unread,
Sep 20, 2016, 8:29:49 PM9/20/16
to sta...@clarkparsia.com
On Tue, Sep 20, 2016 at 7:26 PM, <charbe...@gmail.com> wrote:
 Thank you Jess and Zachary,

@Zachary: I will try this query to check if it makes any difference.

@Jess: I would prefer not to import the virtual graph into Stardog as it is getting updated every 5 min. But will try it to have an idea about performance.
I guess with the materialization operation copies the data from the RDBMS and indexes it locally in Stardog?


Yes, that's correct.

Jess
 
You received this message because you are subscribed to a topic in the Google Groups "Stardog" group.
To unsubscribe from this topic, visit https://groups.google.com/a/clarkparsia.com/d/topic/stardog/J00Fc89Q5aw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stardog+unsubscribe@clarkparsia.com.

Charbel Kaed

unread,
Sep 21, 2016, 4:00:21 PM9/21/16
to Stardog
@Zachary: I tried the query you suggested, it still the same timeout query.

@Jess: I started the virtual graph import,

According to [1], 1 billion triples would require 4GB of JVM memory and 8GB of off-heap memory.
in my RDBMS I have 50 million records, every record has 3 elements a, b, c with a and b as primary keys.
so I assumed that every record will generate the following
a hasBlankNode _:bn#someID,
_:bn#someID hasR2RMLProperty1 b,
_:bn#someID hasR2RMLProperty c,
Which means for every record in my RDBMS I will have 3 triples.
Which makes the total 50 million records x 3 triples = 150 million triples.

any hints to how much heap mem and jvm mem would be required? 


With the following configuration, I have the following error: 


STARDOG_JAVA_ARGS="-Xmx5g -Xms5g"

./stardog-admin virtual import BOC-SBOXVSFD-SDF2 semantic.properties semantic.ttl
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000755580000, 1789394944, 0) failed; error='Cannot allocate memory' (errno=12)
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 1789394944 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/osadmin/stardog-4.1.3/bin/hs_err_pid42821.log

Thank you,
 

Zachary Whitley

unread,
Sep 21, 2016, 4:11:14 PM9/21/16
to Stardog
Like Jess mentioned, the query optimizer is going to generate the same query. I didn't know what exactly the query optimizer was having a hard time with. (I'm guessing it has to do with not being able to generate cardinality estimates over logical tables)

How much ram do you have on your server? It looks like you didn't specify an off heap size -XX:MaxDirectMemorySize=???

I'm not sure exactly what it would do in that case. I'd guess it would use the default.

--

Jess Balint

unread,
Sep 21, 2016, 4:23:09 PM9/21/16
to sta...@clarkparsia.com
Can you share this crash log? How much total memory is available on your system?

Jess
 
Thank you,
 

Zachary Whitley

unread,
Sep 21, 2016, 4:25:49 PM9/21/16
to Stardog
I just noticed that he's running on a VM. I have seen similar issues while running on a VM but unfortunately I'm unable to share the logs. :(

Charbel Kaed

unread,
Sep 21, 2016, 4:51:50 PM9/21/16
to Stardog
@Zachary was right, I specified the 
 -XX:MaxDirectMemorySize and it worked.


We are so close :)

I am having another issue now with the import.

I did this first:
 ./stardog-admin virtual add --format r2rml semantic.properties semantic.ttl

then this:
 ./stardog-admin virtual import --format r2rml BOC-SBOXVSFD-SDF semantic.properties semantic.ttl


I have got this error: 
"Too many pattern letters: a"

I am not sure what it means?

Here is my r2rml file:



<SQLDBTriplesMap>
    a rr:TriplesMap;
    rr:logicalTable [ rr:tableName "sqldb4" ];
    rr:subjectMap [ rr:template "http://www.mycompany.com/Clients/BoC#{\"id\"}" ;
                    rr:class qt:Elements ];
    rr:predicateObjectMap [
      rr:predicate boc:id ;
      rr:objectMap    [ rr:column "\"id\""; rr:datatype xsd:string ]
    ];
rr:predicateObjectMap [
      rr:predicate qt:hasparam1 ;
      rr:objectMap    [ rr:column "\"param1\""; rr:datatype xsd:double ]
    ];
    rr:predicateObjectMap [
      rr:predicate qt:hasparam2 ;
      rr:objectMap [ rr:column "\"param2\""; rr:datatype xsd:double ]
    ].

Jess Balint

unread,
Sep 21, 2016, 5:09:39 PM9/21/16
to sta...@clarkparsia.com
On Wed, Sep 21, 2016 at 3:51 PM, Charbel Kaed <charbe...@gmail.com> wrote:
@Zachary was right, I specified the 
 -XX:MaxDirectMemorySize and it worked.


We are so close :)


Great.
 
I am having another issue now with the import.

I did this first:
 ./stardog-admin virtual add --format r2rml semantic.properties semantic.ttl

then this:
 ./stardog-admin virtual import --format r2rml BOC-SBOXVSFD-SDF semantic.properties semantic.ttl


I have got this error: 
"Too many pattern letters: a"

I am not sure what it means?


This is an issue with VG materialization from SQL Server that is fixed in Stardog 4.2 which we plan to release around the middle of next week. In the meantime, you can disable the import optimizations leading to this error by placing "import.optimize=false" in your properties file.

Jess
 
Here is my r2rml file:



<SQLDBTriplesMap>
    a rr:TriplesMap;
    rr:logicalTable [ rr:tableName "sqldb4" ];
    rr:subjectMap [ rr:template "http://www.mycompany.com/Clients/BoC#{\"id\"}" ;
                    rr:class qt:Elements ];
    rr:predicateObjectMap [
      rr:predicate boc:id ;
      rr:objectMap    [ rr:column "\"id\""; rr:datatype xsd:string ]
    ];
rr:predicateObjectMap [
      rr:predicate qt:hasparam1 ;
      rr:objectMap    [ rr:column "\"param1\""; rr:datatype xsd:double ]
    ];
    rr:predicateObjectMap [
      rr:predicate qt:hasparam2 ;
      rr:objectMap [ rr:column "\"param2\""; rr:datatype xsd:double ]
    ].

--

Charbel Kaed

unread,
Sep 22, 2016, 11:03:38 AM9/22/16
to Stardog
Hello,

I tried the import but got timeout, the performance of the SQL where 100% CPU while doing the import for about an hour then I got timeout exception.

I resized my Linux VM which is running nothing but Stardog to 8 cores and 56 GB RAM and 800 GB SSD. Any recommendation on the JVM and Off Memory heap size?

Thank you,

Zachary Whitley

unread,
Sep 22, 2016, 11:14:41 AM9/22/16
to Stardog
What exactly did you run to do the import?

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+unsubscribe@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

Charbel Kaed

unread,
Sep 22, 2016, 11:22:03 AM9/22/16
to Stardog
I first created the virtual graph:

 ./stardog-admin virtual add --format r2rml semantic.properties semantic.ttl

then import it?

Charbel Kaed

unread,
Sep 22, 2016, 11:23:33 AM9/22/16
to Stardog
This is the charts on the SQL Azure side:
Capture.PNG

Jess Balint

unread,
Sep 22, 2016, 12:08:21 PM9/22/16
to sta...@clarkparsia.com
It doesn't appear to be a memory issue at this point. What do you see in the log when the timeout occurs?

Jess

Charbel Kaed

unread,
Sep 22, 2016, 1:42:33 PM9/22/16
to Stardog
Hello Jess,

I got this error message: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data

in the log got the following:

INFO  2016-09-22 16:31:51,017 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(184): Initializing R2RML registry
INFO  2016-09-22 16:31:51,957 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(193): Loaded R2RML registry with 3 sources
INFO  2016-09-22 16:31:52,544 [main] com.complexible.stardog.StardogKernel:start(1864): Initializing Stardog
WARN  2016-09-22 16:31:52,567 [main] com.complexible.stardog.metadata.MetadataIO:readBytes(260): Unrecognized option in the metadata: search.version
WARN  2016-09-22 17:09:08,582 [Stardog.Executor-3] com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl:varType(442): Cannot determine the type of the variable in the update query: null
ERROR 2016-09-22 17:16:13,266 [Stardog.Executor-3] com.complexible.tx.api.impl.DefaultTransaction:computePrepareResult(489): There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b
com.complexible.tx.api.ResourceTransactionException: There was an error while indexing the new data
at com.complexible.stardog.search.db.ConnectableSearchConnectionImpl$SearchIndexResourceTransaction.prepare(ConnectableSearchConnectionImpl.java:410) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.tx.api.BaseResourceTransaction.prepare(BaseResourceTransaction.java:187) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.computePrepareResult(DefaultTransaction.java:473) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.runPreparePhase(DefaultTransaction.java:439) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.commit(DefaultTransaction.java:330) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:405) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:87) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:165) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3708) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3695) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:510) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:149) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:338) ~[stardog-protocols-api-server-4.1.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: com.complexible.stardog.search.IndexingException: /tmp/1474561960287-0/sd1474561960289-0 (No such file or directory)
at com.complexible.stardog.search.waldo.DefaultIndexer.index(DefaultIndexer.java:209) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:134) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:108) ~[stardog-search-waldo-4.1.3.jar:?]
... 4 more
Caused by: com.complexible.stardog.plan.eval.operator.OperatorException: /tmp/1474561960287-0/sd1474561960289-0 (No such file or directory)
at com.complexible.stardog.plan.eval.operator.util.DiskSolutionSequence.iterator(DiskSolutionSequence.java:99) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.DiskSolutionSequence.iterator(DiskSolutionSequence.java:34) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl$CachedStmtSource.createOperator(ModifyOperatorImpl.java:275) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl$AbstractStmtSource.statements(ModifyOperatorImpl.java:306) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.search.waldo.DefaultIndexer.index(DefaultIndexer.java:168) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:134) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:108) ~[stardog-search-waldo-4.1.3.jar:?]
... 4 more
Caused by: java.io.FileNotFoundException: /tmp/1474561960287-0/sd1474561960289-0 (No such file or directory)
at java.io.FileInputStream.open0(Native Method) ~[?:1.8.0_91]
at java.io.FileInputStream.open(FileInputStream.java:195) ~[?:1.8.0_91]
at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[?:1.8.0_91]
at com.complexible.stardog.plan.eval.operator.util.DiskSolutionSequence.iterator(DiskSolutionSequence.java:96) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.DiskSolutionSequence.iterator(DiskSolutionSequence.java:34) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl$CachedStmtSource.createOperator(ModifyOperatorImpl.java:275) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl$AbstractStmtSource.statements(ModifyOperatorImpl.java:306) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.search.waldo.DefaultIndexer.index(DefaultIndexer.java:168) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:134) ~[stardog-search-waldo-4.1.3.jar:?]
at com.complexible.stardog.search.IndexingStrategy$RunIndexer.call(IndexingStrategy.java:108) ~[stardog-search-waldo-4.1.3.jar:?]
... 4 more
ERROR 2016-09-22 17:16:13,287 [Stardog.Executor-3] com.complexible.stardog.db.DatabaseConnectionImpl:commit(414): There was an error committing the transaction, all changes were rolled back successfully
com.complexible.tx.api.HeuristicRollbackException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.tx.api.impl.DefaultTransaction.runPreparePhase(DefaultTransaction.java:444) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.commit(DefaultTransaction.java:330) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:405) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:87) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:165) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3708) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3695) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:510) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:149) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:338) ~[stardog-protocols-api-server-4.1.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
ERROR 2016-09-22 17:16:13,288 [Stardog.Executor-3] com.complexible.stardog.protocols.server.SPECServerFunction:query(520): There was an error executing query: ADD <virtual://989927c0-d0f2-4c43-92ad-1e99278cdff9> TO <tag:stardog:api:context:default>
com.complexible.stardog.plan.eval.operator.OperatorException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:106) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:87) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:165) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3708) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3695) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:510) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:149) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:338) ~[stardog-protocols-api-server-4.1.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: com.complexible.stardog.db.DatabaseException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:415) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
... 25 more
Caused by: com.complexible.tx.api.HeuristicRollbackException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.tx.api.impl.DefaultTransaction.runPreparePhase(DefaultTransaction.java:444) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.commit(DefaultTransaction.java:330) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:405) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
... 25 more
ERROR 2016-09-22 17:16:13,289 [Stardog.Executor-3] com.complexible.common.protocols.server.rpc.ServerHandler:exceptionCaught(415): exceptionCaughtServerHandler
com.complexible.stardog.plan.eval.operator.OperatorException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:106) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:87) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.1.3.jar:?]
at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.1.3.jar:?]
at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:165) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3708) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3695) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:510) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:149) ~[stardog-protocols-spec-server-4.1.3.jar:?]
at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:338) ~[stardog-protocols-api-server-4.1.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_91]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_91]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_91]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: com.complexible.stardog.db.DatabaseException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:415) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
... 25 more
Caused by: com.complexible.tx.api.HeuristicRollbackException: There was a fatal failure during preparation of d66f2490-49e0-4801-ba77-1eff6cc15a9b There was an error while indexing the new data
at com.complexible.tx.api.impl.DefaultTransaction.runPreparePhase(DefaultTransaction.java:444) ~[stardog-4.1.3.jar:?]
at com.complexible.tx.api.impl.DefaultTransaction.commit(DefaultTransaction.java:330) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:405) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.db.DelegatingTransactionalConnectableConnection.commit(DelegatingTransactionalConnectableConnection.java:68) ~[stardog-4.1.3.jar:?]
at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:89) ~[stardog-4.1.3.jar:?]
... 25 more

Charbel Kaed

unread,
Sep 22, 2016, 2:17:43 PM9/22/16
to sta...@clarkparsia.com
SELECT 
   1 AS "subjectQuestType", NULL AS "subjectLang", (CAST ('http://www.mycompany.com/Clients/BoC#' as varchar(8000)) + CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(QVIEW1."id", ' ', '%20'), '!', '%21'), '''', '%22'), '#', '%23'), '$', '%24'), '&', '%26'), '(', '%28'), ')', '%29'), '*', '%2A'), '+', '%2B'), ',', '%2C'), '/', '%2F'), ':', '%3A'), ';', '%3B'), '=', '%3D'), '?', '%3F'), '@', '%40'), '[', '%5B'), ']', '%5D') as varchar(8000))) AS "subject", 
   1 AS "predicateQuestType", NULL AS "predicateLang", 'http://www.mycompany.com/global/CoreSE_Quantity#hasParam1' AS "predicate", 
   8 AS "objectQuestType", NULL AS "objectLang", CAST(QVIEW1."t" AS VARCHAR(8000)) AS "object"
 FROM 
"dbo"."mydb4" QVIEW1
WHERE 
QVIEW1."id" IS NOT NULL AND
QVIEW1."t" IS NOT NULL
UNION ALL
SELECT 
   1 AS "subjectQuestType", NULL AS "subjectLang", (CAST ('http://www.mycompany.com/Clients/BoC#' as varchar(8000)) + CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(QVIEW1."id", ' ', '%20'), '!', '%21'), '''', '%22'), '#', '%23'), '$', '%24'), '&', '%26'), '(', '%28'), ')', '%29'), '*', '%2A'), '+', '%2B'), ',', '%2C'), '/', '%2F'), ':', '%3A'), ';', '%3B'), '=', '%3D'), '?', '%3F'), '@', '%40'), '[', '%5B'), ']', '%5D') as varchar(8000))) AS "subject", 
   1 AS "predicateQuestType", NULL AS "predicateLang", 'http://www.mycompany.com/Clients/BoC#id' AS "predicate", 
   7 AS "objectQuestType", NULL AS "objectLang", QVIEW1."id" AS "object"
 FROM 
"dbo"."mydb4" QVIEW1
WHERE 
QVIEW1."id" IS NOT NULL
UNION ALL
SELECT 
   1 AS "subjectQuestType", NULL AS "subjectLang", (CAST ('http://www.mycompany.com/Clients/BoC#' as varchar(8000)) + CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(QVIEW1."id", ' ', '%20'), '!', '%21'), '''', '%22'), '#', '%23'), '$', '%24'), '&', '%26'), '(', '%28'), ')', '%29'), '*', '%2A'), '+', '%2B'), ',', '%2C'), '/', '%2F'), ':', '%3A'), ';', '%3B'), '=', '%3D'), '?', '%3F'), '@', '%40'), '[', '%5B'), ']', '%5D') as varchar(8000))) AS "subject", 
   1 AS "predicateQuestType", NULL AS "predicateLang", 'http://www.mycompany.com/global/CoreSE_Quantity#hasParam2' AS "predicate", 
   6 AS "objectQuestType", NULL AS "objectLang", CAST(QVIEW1."param2" AS VARCHAR(8000)) AS "object"
 FROM 
"dbo"."mydb4" QVIEW1
WHERE 
QVIEW1."id" IS NOT NULL AND
QVIEW1."param2" IS NOT NULL
UNION ALL
SELECT 
   1 AS "subjectQuestType", NULL AS "subjectLang", (CAST ('http://www.mycompany.com/Clients/BoC#' as varchar(8000)) + CAST(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(QVIEW1."id", ' ', '%20'), '!', '%21'), '''', '%22'), '#', '%23'), '$', '%24'), '&', '%26'), '(', '%28'), ')', '%29'), '*', '%2A'), '+', '%2B'), ',', '%2C'), '/', '%2F'), ':', '%3A'), ';', '%3B'), '=', '%3D'), '?', '%3F'), '@', '%40'), '[', '%5B'), ']', '%5D') as varchar(8000))) AS "subject", 
   1 AS "predicateQuestType", NULL AS "predicateLang", 'http://www.w3.org/1999/02/22-rdf-syntax-ns#type' AS "predicate", 
   1 AS "objectQuestType", NULL AS "objectLang", 'http://www.mycompany.com/global/CoreSE_Quantity#Point' AS "object"
 FROM 
"dbo"."mydb4" QVIEW1
WHERE 
QVIEW1."id" IS NOT NULL

Evren Sirin

unread,
Sep 22, 2016, 2:39:44 PM9/22/16
to Stardog
The error message is coming from the the search (full-text) index
saying a temporary file that it created is now missing. Either there
is a bug in Stardog that cleans these files too eagerly or your VM is
setup to clean the temp dir eagerly. We'll try to reproduce this on
our side. Easiest workaround for you would be to import data into
Stardog with search disabled and then turn on search.

Best,
Evren

[1] http://docs.stardog.com/#_configuring_temporary_scratch_space

On Thu, Sep 22, 2016 at 2:17 PM, Charbel Kaed <charbe...@gmail.com> wrote:
> Here is the SQL Query received on MS SQL Azure. Hope if helps.
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com

Charbel Kaed

unread,
Sep 22, 2016, 3:48:13 PM9/22/16
to Stardog
Thank you Evren, Jess and Zachary !
I disabled the search and it is working now. I imported 50 000 000 record.

Best,
Charbel 

Pavel Klinov

unread,
Sep 26, 2016, 5:23:19 AM9/26/16
to sta...@clarkparsia.com
Charbel,

We've been able to reproduce the issue with the search index (#3162 in the internal tracker) and are working on it.

Thanks
Pavel

--

Charbel Kaed

unread,
Sep 26, 2016, 10:54:13 AM9/26/16
to Stardog
Hello,

I noticed I had an error in my ttl file. And I needed BlankNodes.

Since, I am getting GC overheads or Stardog don't respond anymore. I set the parameters to the following: STARDOG_JAVA_ARGS="-Xms16g -Xmx16g -XX:MaxDirectMemorySize=32g" on an 8 cores linux VM with 56 GB of RAM and 112 GM of SSD.
 
I attached the ttl file and what the SQL parameters. I am not an expert in R2RML. I hope I am not doing some not recommended mapping.
The import workes well on a 10 records DB. 

Thank you,
Charbel
semantic.ttl
Picture2.png

Evren Sirin

unread,
Sep 27, 2016, 9:55:27 AM9/27/16
to Stardog
Hi Charbel,

There are some problems with bnodes in mappings that we are aware of
(open ticket number is 2775). Do you really have to use a bnode here?
Why not turn it into a IRI by prefixing that template with a
namespace?

Best,
Evren
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com

Charbel Kaed

unread,
Sep 27, 2016, 12:01:02 PM9/27/16
to Stardog
Hello Evren,

Thank you for your reply,
I was not aware for the open ticket.

I agree with you, I was not using BlankNodes with the virtual graph creation. 
My R2RML file is as follows:

<SeriesTriplesMap>
    a rr:TriplesMap;
    rr:logicalTable [ rr:tableName "testdb" ];
    rr:subjectMap [ rr:template "http://www.mycompany.com/Clients/BoC#{\"seriesid\"}" ;
                    rr:class qt:Series ];
    rr:predicateObjectMap [
      rr:predicate boc:id ;
      rr:objectMap    [ rr:column "\"id\""; rr:datatype xsd:string ]
    ];
rr:predicateObjectMap [
      rr:predicate qt:hasParameter1 ;
      rr:objectMap    [ rr:column "\"param1\""; rr:datatype xsd:dateTime ]
    ];
    rr:predicateObjectMap [
      rr:predicate qt:hasParameter2 ;
      rr:objectMap [ rr:column "\"param2\""; rr:datatype xsd:double ]
    ].

However, when I used the same file for import, the structure was flat. 
I have three columns in MS SQL
Id, param1, param2 with id and param1 as primary keys. So I needed the blank nodes for the materialisation ?

Evren, can you please provide some guidelines on how to do that what you are proposing? I am still a beginner in the R2RML.

Thank you, 

Jess Balint

unread,
Sep 27, 2016, 12:54:30 PM9/27/16
to sta...@clarkparsia.com
Your object map is specified as:

        rr:template "Param1_2_Node{seriesid}_{t}";
        rr:termType rr:BlankNode;

You specifically request the generated object to be a blank node. If you use "rr:IRI" instead you can change the template to include a namespace, e.g. 

        rr:template "http://www.mycompany.com/Clients/Param1_2_Node{seriesid}_{t}";
        rr:termType rr:IRI;

Hope this helps.
Jess


--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to

charbel kaed

unread,
Oct 3, 2016, 11:24:04 AM10/3/16
to sta...@clarkparsia.com
Hello Jess,

Thank you, I will try it on the 4.2 version.



---
You received this message because you are subscribed to a topic in the Google Groups "Stardog" group.
To unsubscribe from this topic, visit https://groups.google.com/a/clarkparsia.com/d/topic/stardog/J00Fc89Q5aw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stardog+unsubscribe@clarkparsia.com.

Charbel Kaed

unread,
Oct 5, 2016, 12:20:08 AM10/5/16
to Stardog
Hello,

I let go the materialisation and going for the virtual graph.
I would like to have your opinion on the following behaviour.

Context:
I am relying on a MS SQL with the schema as described in the attachment. SeriesId (primary key). param1 (primary key), param2.

For the R2RML part, I suggest the following:


<SeriesTriplesMap>
    a rr:TriplesMap;
    rr:logicalTable [ rr:tableName "testdb" ];
    rr:subjectMap [ rr:template "http://www.mycompany.com/Clients/BoC#{\"seriesid\"}" ;
                    rr:class qt:Series ];
    rr:predicateObjectMap [
      rr:predicate  boc:id ;
      rr:objectMap    [ rr:column "\"id\""; rr:datatype xsd:string ]
    ];
rr:predicateObjectMap [
      rr:predicate  qt:hasParameter1 ;
      rr:objectMap    [ rr:column "\"param1\""; rr:datatype xsd:dateTime ]
    ];
    rr:predicateObjectMap [
      rr:predicate qt:hasParameter2 ;
      rr:objectMap [ rr:column "\"param2\""; rr:datatype xsd:double ]
    ].

In Stardog when I create a virtual graph without materialisation. 

Queries:
When I request in Sparql query the (seriesId and param1) or (seriesId and param2), the execution time is around 9-10 minutes. 
However, when I request the param1 and param2 then Stardog stays more than an hour to get results. And eventually the timeout occurs and either the query drops or Stardog becomes freezing and non-interactive.

Do you have an idea how can I get the param1 and param2 in a single query? Am I having something wrong in the R2RML file?

Thank you for your help,

Cheers,
Charbel.

Picture2 (1).png

Evren Sirin

unread,
Oct 5, 2016, 9:13:14 AM10/5/16
to Stardog
Can you send us the output of `stardog query explain` command for both
queries? The output should show both the query string and the query
plan.

Thanks,
Evren
> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com

Charbel Kaed

unread,
Oct 14, 2016, 2:08:45 PM10/14/16
to Stardog
Hello,
 
I am trying with Stardog 4.2 to materialize the graph.

I am having the following error, any idea how i should handle it?

Thank you, 

ERROR 2016-10-14 17:48:16,079 [StardogServer.WorkerGroup-3] com.complexible.stardog.protocols.http.server.HttpMessageEncoder:createErrorResponse(288): The result encoder received an error message it could not encode, error was:
com.complexible.stardog.plan.eval.operator.OperatorException: java.lang.IllegalArgumentException
        at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl.computeNext(ModifyOperatorImpl.java:163) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl.computeNext(ModifyOperatorImpl.java:79) ~[stardog-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.jar:?]
        at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
        at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:85) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.impl.UpdateSequenceOperatorImpl.computeNext(UpdateSequenceOperatorImpl.java:29) ~[stardog-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:115) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.util.AutoCloseOperator.computeNext(AutoCloseOperator.java:25) ~[stardog-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.tryToComputeNext(AbstractSkippingIterator.java:143) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.collect.AbstractSkippingIterator.hasNext(AbstractSkippingIterator.java:130) ~[stardog-utils-common-4.2.jar:?]
        at com.google.common.collect.Iterators.size(Iterators.java:191) ~[guava-18.0.jar:?]
        at com.complexible.stardog.plan.eval.QueryEngine.executeUpdate(QueryEngine.java:166) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:263) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.query.DefaultQueryFactory$UpdateQueryImpl.execute(DefaultQueryFactory.java:242) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1571) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.db.DatabaseImpl$TimerPauseUpdate.execute(DatabaseImpl.java:1559) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.query.DelegatingUpdateQuery.execute(DelegatingUpdateQuery.java:139) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3772) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.StardogKernel$SecuredUpdateQuery.execute(StardogKernel.java:3759) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.protocols.server.SPECServerFunction.query(SPECServerFunction.java:526) ~[stardog-protocols-spec-server-4.2.jar:?]
        at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:152) ~[stardog-protocols-spec-server-4.2.jar:?]
        at com.complexible.common.protocols.server.rpc.ServerHandler.lambda$handleMessage$1(ServerHandler.java:336) ~[stardog-protocols-api-server-4.2.jar:?]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.lang.IllegalArgumentException
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source) ~[?:?]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_101]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_101]
        at com.complexible.common.memory.PlatformDependent0$BitsJDK7Or8.reserveMemory(PlatformDependent0.java:597) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.Memory.malloc(Memory.java:43) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.MemoryPools$MemoryPoolImpl.malloc(MemoryPools.java:390) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.MemoryPools$MemoryPoolImpl.calloc(MemoryPools.java:294) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.OffHeap.calloc(OffHeap.java:153) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.OffHeapObjectArray.create(OffHeapObjectArray.java:100) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.OffHeapArrays.copyOf(OffHeapArrays.java:54) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.OffHeapArrayList.ensureSize(OffHeapArrayList.java:323) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.common.memory.OffHeapArrayList.add(OffHeapArrayList.java:110) ~[stardog-utils-common-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.util.UnsafeSolutionSequence$Builder.add(UnsafeSolutionSequence.java:110) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.CopyingSolutionSequenceBuilders$SequenceBuilder.copyFrom(CopyingSolutionSequenceBuilders.java:22) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.FlowingSolutionSequences$FlowingAcrossStorageBuilder.copyFromPrevious(FlowingSolutionSequences.java:340) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.FlowingSolutionSequences$FlowingAcrossStorageBuilder.flowToNext(FlowingSolutionSequences.java:348) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.FlowingSolutionSequences$FlowingAcrossStorageBuilder.flowOffHeap(FlowingSolutionSequences.java:326) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.FlowingSolutionSequences$FlowingAcrossStorageBuilder.add(FlowingSolutionSequences.java:308) ~[stardog-4.2.jar:?]
        at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl.lambda$computeNext$0(ModifyOperatorImpl.java:146) ~[stardog-4.2.jar:?]
        at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[?:1.8.0_101]
        at com.complexible.stardog.plan.eval.operator.impl.ModifyOperatorImpl.computeNext(ModifyOperatorImpl.java:146) ~[stardog-4.2.jar:?]
        ... 30 more


Evren Sirin

unread,
Oct 14, 2016, 3:21:15 PM10/14/16
to Stardog
It looks like you are using an INSERT/WHERE query for materialization.
Are you trying to materialize only some part of the virtual graph? The
recommended way for materialization is using COPY or ADD [1] which
would be faster even if you bring in more data.

Best,
Evren

[1] http://docs.stardog.com/#_materializing_virtual_graphs

charbel kaed

unread,
Oct 14, 2016, 3:36:53 PM10/14/16
to sta...@clarkparsia.com
Hello Evren

I am using the following to materialize the 50 million records.

 ./stardog-admin virtual import --format r2rml BOC-SBOXVSFD-SDF semantic.properties sem.ttl

Best,
Charbel.



> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
---
Reply all
Reply to author
Forward
0 new messages