Database does not exist !

[c.sip...@gmail.com]

Hi,

* I just used Stardog to make tests, I loaded 66 136 542 triples from RDFXML files using command line tool.

 ./bin/stardog query mybase "select (count(*) as ?c) {?s ?p ?o}"
+----------+
|    c     |
+----------+
| 66136542 |
+----------+

* Then I stopped the server:

./bin/stardog-admin server stop
Stardog server successfully received the shutdown request.

* I restarted it:

./bin/stardog-admin server start
Initializing Stardog
Database mybase will not be available because there was an error initializing the database: java.lang.ArrayIndexOutOfBoundsException
Moving irreparable database mybase to /var/lib/stardog/data/.unusable/mybase

* So, now my data are not available! 
Is there something to do?

Clement

[Michael Grove]

On Mon, Dec 14, 2015 at 9:52 AM, <c.sip...@gmail.com> wrote:

Hi,

* I just used Stardog to make tests, I loaded 66 136 542 triples from RDFXML files using command line tool.

 ./bin/stardog query mybase "select (count(*) as ?c) {?s ?p ?o}"
+----------+
|    c     |
+----------+
| 66136542 |
+----------+

* Then I stopped the server:

./bin/stardog-admin server stop
Stardog server successfully received the shutdown request.

stop is an async operation.  Did you try to subsequently kill the process?

 

* I restarted it:

./bin/stardog-admin server start
Initializing Stardog
Database mybase will not be available because there was an error initializing the database: java.lang.ArrayIndexOutOfBoundsException
Moving irreparable database mybase to /var/lib/stardog/data/.unusable/mybase

Are there any stack traces in stardog.log?

What configuration did you use to create the database?

Cheers,

Mike

 

* So, now my data are not available! 
Is there something to do?

Clement

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[c.sip...@gmail.com]

Le lundi 14 décembre 2015 16:15:47 UTC+1, Michael Grove a écrit :

On Mon, Dec 14, 2015 at 9:52 AM, <c.sip...@gmail.com> wrote:

Hi,

* I just used Stardog to make tests, I loaded 66 136 542 triples from RDFXML files using command line tool.

 ./bin/stardog query mybase "select (count(*) as ?c) {?s ?p ?o}"
+----------+
|    c     |
+----------+
| 66136542 |
+----------+

* Then I stopped the server:

./bin/stardog-admin server stop
Stardog server successfully received the shutdown request.

stop is an async operation.  Did you try to subsequently kill the process?

No, and I looked processes with "top" and there was no more Stardog process.

 

* I restarted it:

./bin/stardog-admin server start
Initializing Stardog
Database mybase will not be available because there was an error initializing the database: java.lang.ArrayIndexOutOfBoundsException
Moving irreparable database mybase to /var/lib/stardog/data/.unusable/mybase

Are there any stack traces in stardog.log?

There is lot of trace about malformed XML documents, but I know that a part of my documents are malformed.
After, there is a more interesting line with a memory allocation error, even it looks to be related to RDF parsing there must be a link.

ERROR 2015-12-14 15:19:46,924 [SPEC-Server-1-1] com.complexible.stardog.db.DatabaseConnectionImpl:commit(390): There was an error committing the transaction, all changes were rolled back successfully
com.complexible.tx.api.HeuristicRollbackException: There was a fatal failure during preparation of 253912e1-2ecb-4017-b78e-6110e7cd6e39 org.openrdf.rio.RDFParseException: Unable to allocate 4.1K bytes, direct memory exhausted
        at com.complexible.tx.api.impl.DefaultTransaction.runPreparePhase(DefaultTransaction.java:440) ~[stardog-4.0.2.jar:?]
        at com.complexible.tx.api.impl.DefaultTransaction.commit(DefaultTransaction.java:326) ~[stardog-4.0.2.jar:?]
        at com.complexible.stardog.db.DatabaseConnectionImpl.commit(DatabaseConnectionImpl.java:383) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.db.DatabaseImpl$DBConnectionWrapper.commit(DatabaseImpl.java:1374) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.db.DelegatingDatabaseConnection.commit(DelegatingDatabaseConnection.java:282) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.StardogKernel$KernelDbConnection.commit(StardogKernel.java:2738) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.db.DelegatingDatabaseConnection.commit(DelegatingDatabaseConnection.java:282) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.StardogKernel$NotifyingDatabaseConnection.commit(StardogKernel.java:3047) [stardog-4.0.2.jar:?]
        at com.complexible.stardog.protocols.server.SPECServerFunction.commitTx(SPECServerFunction.java:380) [stardog-protocols-spec-server-4.0.2.jar:?]
        at com.complexible.stardog.protocols.server.SPECServerFunction.handleMessage(SPECServerFunction.java:164) [stardog-protocols-spec-server-4.0.2.jar:?]
        at com.complexible.common.protocols.server.rpc.ServerHandler.handleMessage(ServerHandler.java:324) [stardog-protocols-api-server-4.0.2.jar:?]
        at com.complexible.common.protocols.server.rpc.ServerHandler.channelRead(ServerHandler.java:207) [stardog-protocols-api-server-4.0.2.jar:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:299) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) [netty-all-4.0.32.Final.jar:4.0.32.Final]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_66-internal]
INFO  2015-12-14 15:31:52,212 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(215): Initializing R2RML registry
INFO  2015-12-14 15:31:52,249 [main] com.complexible.stardog.virtual.DefaultVirtualGraphRegistry:syncCache(224): Loaded R2RML registry with 0 sources
Initializing Stardog
WARN  2015-12-14 15:31:53,006 [main] com.complexible.stardog.db.DatabaseFactoryImpl:read(134): Database qualinca is invalid and the repair failed: 8176
Database qualinca will not be available because there was an error initializing the database: java.lang.ArrayIndexOutOfBoundsException
Moving irreparable database qualinca to /var/lib/stardog/data/.unusable/qualinca

What configuration did you use to create the database?

I used a default configuration, I followed the quick start guide (http://docs.stardog.com/#_linux_and_osx). Perhaps I need a specific configuration in order to load this quantity of triples?

for information:
I run Stardog over a
4 cores (8 threads) Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
with 16Go of RAM
and a 180Go SSD

Thanks,

Clement

[Michael Grove]

Are you adding the data using `data add` from the CLI?

Can you give the complete sequence of steps you executed that triggered the issue?

 

What configuration did you use to create the database?

I used a default configuration, I followed the quick start guide (http://docs.stardog.com/#_linux_and_osx). Perhaps I need a specific configuration in order to load this quantity of triples?

for information:
I run Stardog over a
4 cores (8 threads) Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
with 16Go of RAM
and a 180Go SSD

General capacity planning guidelines can be found in the docs [1]

Cheers,

Mike

[1] http://docs.stardog.com/#_capacity_planning

[c.sip...@gmail.com]

yes

Can you give the complete sequence of steps you executed that triggered the issue?

Here are the complete sequence:

// I installed a new developer version

$ ./bin/stardog-admin server start
$ ./bin/stardog-admin db create -n myBase
// I loaded lot of files with the command below (some of files were malformed)

$ for dir in /home/data/sudoc/notices/*/*; 
       do ./bin/stardog data add --format RDFXML myBase $(ls $dir/*);
   done  > log.txt 2>&1 &
// I waited approximately 2 days
// During the loading period, I sent some sparql queries through the web interface and I got correct answers in good response time.
// I stopped the server
$ ./stardog-admin server stop
// I waited there was no more Stardog process in "top"
// I restart the server
$ ./stardog-admin server start
// And I got this issue

Initializing Stardog
Database mybase will not be available because there was an error initializing the database: java.lang.ArrayIndexOutOfBoundsException
Moving irreparable database mybase to /var/lib/stardog/data/.unusable/mybase

What configuration did you use to create the database?

I used a default configuration, I followed the quick start guide (http://docs.stardog.com/#_linux_and_osx). Perhaps I need a specific configuration in order to load this quantity of triples?

for information:
I run Stardog over a
4 cores (8 threads) Intel(R) Xeon(R) CPU E5-1620 0 @ 3.60GHz
with 16Go of RAM
and a 180Go SSD

General capacity planning guidelines can be found in the docs [1]

Cheers,

Mike

[1] http://docs.stardog.com/#_capacity_planning

 

thanks
 

[Michael Grove]

It took two days to load 66M triples?

 

// During the loading period, I sent some sparql queries through the web interface and I got correct answers in good response time.

Do you need to query while the load is occurring?

Is your data public, or are you otherwise able to share it?  I loaded 100M triples into a server with the default settings and I didnt have any errors and the load completed in about 10m. 

Cheers,

Mike

[c.sip...@gmail.com]

No, but I launched the process the friday and I came back the monday however the loading took more than 10m.
In my logs, I launched it Friday at 4pm and the last trace is Saturday at 6.30pm, so more 1 day.
I think, it's because my data are in 3 429 242 little files.

No, I don't need to query while the load is occuring. It was just for test some queries over data already load.
Sorry, but my data is not public.

I will retry the process and keep you informed.

Regards,
Clement