how to limit stardog memory usage

[Tom Geudens]

Hello,

I'm running Stardog 2.1.3 on a 16g instance on Ubuntu 12.04 LTS. Stardog is started with these parameters :
export STARDOG_JAVA_ARGS="-Xmx4g -Xms4g -XX:MaxDirectMemorySize=8g", java version is 1.7.0_55

Two times already a (faulty) query has been launched against the database (+/- 74 000 000 triples) that makes the Stardog java process consume much much more memory than the parameters would seem to allow. Finally the system does an OOM and kills the database just before Stardog has 16g of RES memory.

Is there a way - or a set of parameters - that would prevent Stardog from using memory beyond what the parameters allow it to use ?

Regards,
Tom Geudens

[Mike Grove]

It should be respecting your direct memory setting, the off-heap pool is based on the size specified in the JVM args.  Is there a way we could see the data & query to reproduce locally?

You can try adding the JVM parameter "-Denable.unsafe=false" to disable use of off-heap memory, which if there's an error in the pool size calculation, could avoid the problem.

Cheers,

Mike 

 

Regards,
Tom Geudens

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Tom Geudens]

Hello Mike,

Well, there's a full dump available at data.http://kbodata.be/download/kbo.zip
The query is as follows :
CONSTRUCT {
<http://data.kbodata.be/organisation/0200_065_765#id> ?p ?v.
?v ?p ?o.
?v ?pp ?vv.
?vv ?ppp ?vvv.
}
WHERE {
<http://data.kbodata.be/organisation/0200_065_765#id> ?p ?v.
?v ?pp ?vv.
?vv ?ppp ?vvv.
}

As you can see the ?v ?p ?o. line is faulty, but the query starts executing anyway and eats memory. I tried adding -Denable.unsafe=false as you suggested, but it makes no difference, the process goes right up to the available 16g RES and then gets killed by the system.

Regards,
Tom 

[Mike Grove]

Tom,

Thanks for sending this along.  I was able to pinpoint the issue to an unfortunate interaction between two of our optimizers and we ended up with a suboptimal plan, as you noticed!

With the fix in place, I'm able to run that query in a few ms with the default Stardog settings.

This will be in the next release.

Cheers,

Mike

[Tom Geudens]

Mike,

You're most welcome. Looking forward to the next release !

Regards,
Tom