bulk loading errors

[Ric Roberts]

Hi. We've encountered some errors loading a directory of nquads into stardog (version 3.1.4).

The command we used to start the load was:

$STARDOG_DIST/bin/stardog-admin db create -n pmd-data -t D   -o strict.parsing=false   -o query.all.graphs=true   -o reasoning.type=none   --  /sg-load-data/*.nq

...where /sg-load-data is a directory of ~400 nquads files totalling about ~350GB.

Attached is the contents of stardog.log (there are lots of warnings throughout, plus severe errors near the end).

After about 24 hours CPU usage dropped to 0% and no new log items were added. In the past it's only taken a few hours to load this quantity of data.

Any pointers appreciated!

Cheers,

Ric.

[Evren Sirin]

Hi Ric,

Thanks for the report. The issue #2497 is created for this. We'll look
into the exceptions in the log and try to figure out what happened.

Best,
Evren

> --
> -- --
> You received this message because you are subscribed to the C&P "Stardog"
> group.
> To post to this group, send email to sta...@clarkparsia.com
> To unsubscribe from this group, send email to
> stardog+u...@clarkparsia.com
> For more options, visit this group at
> http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Bill Roberts]

Hi Evren - an update on Ric’s report for #2497.

Ric’s errors arose with Java args of 5g heap and 8g direct memory. We re-ran it with 6g heap and 15g direct memory and the same data loaded successfully (about 900 million triples).

So it was probably caused by some kind of out of memory problem.

Cheers

Bill

> To unsubscribe from this group and stop receiving emails from it, send an email to stardog+u...@clarkparsia.com.