Hi all,Long time since I've been in this space, hope you all are doing well :)
We've recently upgraded from openrdf 4.1.2 to rdf4j 3.4.3 and we're now encountering the following NullPointerException:java.lang.NullPointerException: nullat org.eclipse.rdf4j.sail.nativerdf.SailSourceModel.remove(SailSourceModel.java:291)at org.eclipse.rdf4j.sail.nativerdf.MemoryOverflowModel.remove(MemoryOverflowModel.java:134)at org.eclipse.rdf4j.model.impl.AbstractModel.remove(AbstractModel.java:171)
In the issue tracker there is #693 and #774 which are related to this, but both have been closed in earlier releases. We have now upgraded to rdf4j 3.6.2, but the problem remains.The calling code doesn't actually do anything fancy. The relevant part of the algorithm starts a READ_COMMITTED transaction and then calls getStatements(s,p,o), add(s,p,o) and remove(s,p,o) a lot of times. The entire algorithm is single threaded and runs on a simple NativeStore. The process runs fine for 1 to 2 hours, after which these exception start to appear.Is this a known issue? Any suggestions how to fix or where to look? The NativeStore code looks quite different from when I last worked on it...
Not sure if I'll have the time to debug it properly now, but if I find anything I'll definately let you know. Observation so far is that this NPE is triggered for the first time shortly after the model "overflows" to disk, or at least that's my understanding of what happens in the code so far:
* Initially the algorithm starts with a very good speed, but the
memory usage is growing quickly. I guess in this phase everything
is still stored in RAM. This takes about 12 minutes.
* Then when the memory usage gets close to 80 or 90% the process slows down alot. Memory usage stays at this 80-90% but cpu usage goes up a lot. We're not seeing any progress during this time, but not sure if the process has completely stalled or if it's just very slow. This takes about 40 minutes.
* Then after this period the memory usage drops drastically to a steady 1 GB and we start to see progress again. The first NPE is triggered some 90 seconds after this.
Does this help?
--
You received this message because you are subscribed to a topic in the Google Groups "RDF4J Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rdf4j-users/NaPslvPoS6k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rdf4j-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rdf4j-users/1c5b1f77-99c1-4207-8333-400cd265cff8%40beta.fastmail.com.
You received this message because you are subscribed to the Google Groups "RDF4J Users" group.To unsubscribe from this group and stop receiving emails from it, send an email to rdf4j-users...@googlegroups.com.To view this discussion on the web visit https://groups.google.com/d/msgid/rdf4j-users/79ded8de-0614-37d9-deac-1dc050988e16%40gmail.com.
The reason that syncing the data to disk when the overflow is triggered takes so long is also related to a call to dataset(). When overflowToDisk() calls disk.addAll(memory), this triggers a call to SailSourceModel.add(s,p,o,c...) for each statement. This method then calls both contain(s,p,o,c...) and sink().approve(s,p,o.c) for for each statement. The latter call starts a new transaction and updates the txn-status file, but the contains() call then commits the transaction for the previous statement via a call to dataset(), again updating the txn-status file. So for every cached statement, rdf4j does two I/O calls. On a spinning disk with a an average write time of 10 ms, this limits the overflow process to at most 50 statements per second.
To view this discussion on the web visit https://groups.google.com/d/msgid/rdf4j-users/cf13451e-60b4-473d-9f70-fcd93378fff6%40beta.fastmail.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rdf4j-users/106a43d7-1901-1576-2ae6-b59f4d88242f%40gmail.com.