TupleIndex out of memory

[Jim Balhoff]

Hi,

I’m trying to run HermiT on the Mondo disease ontology. I’m using ROBOT to run the reasoner. After a couple of hours the program fails with this message:

"The space of nodes in TupleIndex was exhausted: the ontology is just too large.”

I found that in the source here: https://github.com/phillord/hermit-reasoner/blob/37ec30aced32ac81ebecc5e33fad255ddefcb4c3/src/main/java/org/semanticweb/HermiT/tableau/TupleIndex.java#L261

This error is being thrown (I think) well before the JVM runs out of heap. By looking at processes with `top`, it seems to happen when the heap is around 200 GB. I am running with max heap of 1000 GB. Is there any workaround for this? Can the TupleIndex be configured?

Thank you,
Jim

[Ignazio Palmisano]

On Mon, 14 Nov 2022 at 02:22, Jim Balhoff <bal...@gmail.com> wrote:
>
> Hi,
>
> I’m trying to run HermiT on the Mondo disease ontology. I’m using ROBOT to run the reasoner. After a couple of hours the program fails with this message:
>

Hi, which HermiT build are you using? (i.e., does it use OWLAPI 4 or
5? I'm not sure whether ROBOT is still limited to OWLAPI 4 or has
updated to 5)

I ask because the repository you linked isn't being updated - there
are forks for OWLAPI 4 an 5, here
https://github.com/owlcs/hermit-reasoner - not very active, but I
don't know of any more recent public repository.

Looking at where that exception gets thrown, I believe the issue is
that the error is thrown if the node counter overflows - it's an int,
and if it gets incremented past MAXINT it will become negative.
With 200 or more gigabytes of heap, I guess an int is no longer fit
for purpose. At first look, changing to a long should help, but it
will require a fresh build and release - hence the questions about
versions in use.

Cheers,
I.

> "The space of nodes in TupleIndex was exhausted: the ontology is just too large.”
>
> I found that in the source here: https://github.com/phillord/hermit-reasoner/blob/37ec30aced32ac81ebecc5e33fad255ddefcb4c3/src/main/java/org/semanticweb/HermiT/tableau/TupleIndex.java#L261
>
> This error is being thrown (I think) well before the JVM runs out of heap. By looking at processes with `top`, it seems to happen when the heap is around 200 GB. I am running with max heap of 1000 GB. Is there any workaround for this? Can the TupleIndex be configured?
>
> Thank you,
> Jim
>

> --
> You received this message because you are subscribed to the Google Groups "HermiT Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to hermit-users...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/hermit-users/7F1583A3-E769-407A-9265-BAE7DE987515%40gmail.com.

[Jim Balhoff]

On Nov 14, 2022, at 3:21 PM, Ignazio Palmisano <ipalmisan...@gmail.com> wrote:

On Mon, 14 Nov 2022 at 02:22, Jim Balhoff <bal...@gmail.com> wrote:

Hi,

I’m trying to run HermiT on the Mondo disease ontology. I’m using ROBOT to run the reasoner. After a couple of hours the program fails with this message:

Hi, which HermiT build are you using? (i.e., does it use OWLAPI 4 or
5? I'm not sure whether ROBOT is still limited to OWLAPI 4 or has
updated to 5)

Here’s the artifact used by ROBOT: https://github.com/ontodev/robot/blob/f9a4efa7254d162df42341e551795af7a7c6ad9c/pom.xml#L237-L239

net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413

ROBOT is still on OWLAPI 4; we just stay aligned with Protege.

I ask because the repository you linked isn't being updated - there
are forks for OWLAPI 4 an 5, here
https://github.com/owlcs/hermit-reasoner - not very active, but I
don't know of any more recent public repository.

Looking at where that exception gets thrown, I believe the issue is
that the error is thrown if the node counter overflows - it's an int,
and if it gets incremented past MAXINT it will become negative.
With 200 or more gigabytes of heap, I guess an int is no longer fit
for purpose. At first look, changing to a long should help, but it
will require a fresh build and release - hence the questions about
versions in use.

I might be able to give this a try locally (depending on how many cascading effects that change would bring!). Or else if you are interested I would be glad to try a new build.

Thanks,

Jim

[Ignazio Palmisano]

On Mon, 14 Nov 2022 at 20:29, Jim Balhoff <bal...@gmail.com> wrote:
>
> On Nov 14, 2022, at 3:21 PM, Ignazio Palmisano <ipalmisan...@gmail.com> wrote:
>
> On Mon, 14 Nov 2022 at 02:22, Jim Balhoff <bal...@gmail.com> wrote:
>
>
> Hi,
>
> I’m trying to run HermiT on the Mondo disease ontology. I’m using ROBOT to run the reasoner. After a couple of hours the program fails with this message:
>
>
> Hi, which HermiT build are you using? (i.e., does it use OWLAPI 4 or
> 5? I'm not sure whether ROBOT is still limited to OWLAPI 4 or has
> updated to 5)
>
>
> Here’s the artifact used by ROBOT: https://github.com/ontodev/robot/blob/f9a4efa7254d162df42341e551795af7a7c6ad9c/pom.xml#L237-L239
>
> net.sourceforge.owlapi:org.semanticweb.hermit:1.3.8.413
>
> ROBOT is still on OWLAPI 4; we just stay aligned with Protege.

Branch version4 of the repository linked above is what was used to
build 1.3.8.413 - it is currently at 1.4.5.456, so it's compatible
with OWLAPI 4.5.6, which is what ROBOT is also using. Building an
enhancement on top of that branch seems like the simplest avenue.

As for the fix needed, I'm a bit lost. The index that's overflowing is
an int and an array can't grow larger than an int value, so just
substituting a long won't do. I'm unclear on whether changing the page
size would help - far as I can tell, it'll still try and keep track of
the number of nodes.

Refactoring the whole class to use longs and having a bidimensional
array to back a collection, which can then pretend to have a larger
address space, seems a possibility, but it's quite a bit of work and
I'm worried about increased memory use for smaller ontologies. Also,
verifying that this actually works for heaps larger than 200G and more
than two billion nodes requires more computational resources than I
can muster :-D it's an interesting problem but I don't think I can
provide a solution in a reasonable timeframe.

Only simple thing that I'd try first: there have been a few changes
from 1.3.8 to the current version4, including a few changes that
should have improved speed and memory consumption, so it might be
worth trying a build of the branch as is, in case the behaviour is
significantly different with your ontology.

Cheers,
I.