OBO parser doesn't like something in fly_anatomy_XP.obo

[Nomi Harris]

Hi David,

As you guessed, the problem is not with save per se; it's that the OBO parser is not happy with something in that ontology file.

When you load fly_anatomy_XP.obo, it goes into an infinite loop in getWarning:

Reading [fly_anatomy_XP.obo]

         (allowDangling = false)

         (followImports = true)

Done parsing file

Problem running task:

java.lang.StackOverflowError

        at org.bbop.util.ImprovedHashMap.get(ImprovedHashMap.java:242)

        at org.bbop.util.MultiHashMap.get(MultiHashMap.java:56)

        at org.bbop.util.MultiHashMap.get(MultiHashMap.java:11)

        at org.obo.util.IDUtil.getWarning(IDUtil.java:464)

        at org.obo.util.IDUtil.getWarning(IDUtil.java:468)

        at org.obo.util.IDUtil.getWarning(IDUtil.java:468)

        at org.obo.util.IDUtil.getWarning(IDUtil.java:468)

...

It encounters a similar problem when you try to save.

Here's the relevant code:

                } else if (multiIDMap.containsKey(id)) {

                        Collection<? extends IdentifiedObject> terms = multiIDMap.get(id);

                        Collection<IDResolution> resolutions = new LinkedList<IDResolution>();

                        for (IdentifiedObject io : terms) {

                                boolean isReplacementDangling = session.getObject(io.getID()) == null;

*                                IDWarning rw = getWarning(io.getID(), isReplacementDangling,

                                                session, multiIDMap);

* is line 468.

I will look into this more--I don't have a good understanding of what's going on in getWarning and why it might end up in an infinite loop.  Chris, any thoughts?  David, can you think of anything odd about the IDs in that ontology file that might cause this issue?

By the way, this is not a new OBO/OBO-Edit issue--older versions have the same issue.  I also tried loading previous versions of fly_anatomy_XP.obo and found that 1.11 is fine but 1.12 has the problem.  Unfortunately, there are a large number of differences between those versions, so finding the specific term(s) that cause the problem will not be simple.  http://obo.cvs.sourceforge.net/viewvc/obo/obo/ontology/anatomy/gross_anatomy/animal_gross_anatomy/fly/fly_anatomy_XP.obo?r1=1.12&r2=1.11

Nomi

On Apr 15, 2011, at 4:39 AM, SourceForge.net wrote:

Bugs item #3287434, was opened at 2011-04-15 11:39
Message generated for change (Tracker Item Submitted) made by djs93
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=418257&aid=3287434&group_id=36855

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: OBO-Edit
Group: Mac OS X
Status: Open
Resolution: None
Priority: 7
Private: No
Submitted By: DavidOS (djs93)
Assigned to: Nobody/Anonymous (nobody)
Summary: Will not save

Initial Comment:
Intermittent bug: OBO-Edit 2.1 beta 11 sometimes gets into a state where it will not save.  Dialog windows pop up indicating pre-save verification checks are running, but then nothing happens.

Here's a log extract from an attempted save:

...
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
2011-04-15 12:26:35,979 [Thread-8] ERROR org.bbop.util.AbstractTaskDelegate - Problem running task:
java.lang.StackOverflowError
at java.util.HashMap.getEntry(HashMap.java:344)
at java.util.LinkedHashMap.get(LinkedHashMap.java:280)
at org.obo.datamodel.impl.OBOSessionImpl.getObject(OBOSessionImpl.java:244)
at org.obo.util.IDUtil.getWarning(IDUtil.java:421)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
...

Don't know if significant - but seems to have been unhappy as soon as file was loaded in this session:


2011-04-15 10:01:45,883 [Thread-8] INFO  org.obo.dataadapter.OBOFileAdapter - Reading [/repositories/OC_SVN/branches/Simon/fly_anatomy_XP.obo]
(allowDangling = false)
(followImports = true)
2011-04-15 10:01:46,399 [Thread-8] DEBUG org.obo.dataadapter.OBOFileAdapter - Done parsing file
2011-04-15 10:01:46,697 [Thread-8] ERROR org.bbop.util.AbstractTaskDelegate - Problem running task:
java.lang.StackOverflowError
at java.util.HashMap.getEntry(HashMap.java:344)
at java.util.LinkedHashMap.get(LinkedHashMap.java:280)
at org.obo.datamodel.impl.OBOSessionImpl.getObject(OBOSessionImpl.java:244)
at org.obo.util.IDUtil.getWarning(IDUtil.java:421)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
at org.obo.util.IDUtil.getWarning(IDUtil.java:468)
...  etc

----------------------------------------------------------------------

You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=418257&aid=3287434&group_id=36855

[Chris Mungall]

org.obo.util.getWarning is not cycle-safe:

protected static LinkIDWarning getWarning(Link link, OBOSession session,
MultiMap<String, IdentifiedObject> multiIDMap) {
IDWarning typeWarning = getWarning(link.getType().getID(), TermUtil
.isDangling(link.getType()), session, multiIDMap);
IDWarning parentWarning = getWarning(link.getParent().getID(), TermUtil
.isDangling(link.getParent()), session, multiIDMap);
if (typeWarning == null && parentWarning == null)
return null;
else
return new DefaultLinkIDWarning(link, parentWarning, typeWarning);
}

Cycles over mixed relationship types are perfectly acceptable but older layers within the OE code strata make the opposite assumption.

I believe fly_XP has had these valid cycles for some time, so not sure why this didn't happen before. In this case the triggering event might be any time some warning is triggered involving a term with a cycle. In this case the warning may have been something to do with dangling references.

Suggestions: make a test file that contains cycles and dangling errors, try and replicate and add a junit to ensure future compliance

Traveling next few days, good luck fixing this! Any bug that could potentially lead to data loss and/or states in which the current ontology can't be saved are top priority, we should be proactively prohibiting these

[Nomi Harris]

Although Chris is right that cycles over mixed relationship types could potentially present problems in OE (can someone give me a test ontology for that?), the problem with fly_anatomy_XP turned out to be a tighter loop.

fly_anatomy_XP.obo has a term (FBbt:00001913) whose alt_id is the same as its id. When the obo parser is reading or getting ready to save the ontology, the verification check would look at FBbt:00001913 and then recursively look at its alt_id to make sure it was ok, so it would end up in an infinite loop.
I have fixed this with a two-pronged approach:
1. At the recommendation of the OBO-Edit Working Group, the OBO parser now throws a fatal exception if the alt_id (or replaced_by) is the same as the primary ID. (This means OE will now complain about fly_anatomy_XP.)
2. The place in the verifier where it was going into an infinite loop now checks for that redundancy before recursing. (However, note that it would still be possible to get into a loop if there were a bigger cycle in the ontology--I would appreciate an example of this for testing.)

This fix is in the just-released OE 2.1-b12.

Nomi

[Chris Mungall]

Thanks Nomi!

I haven't analyzed this code enough to know which situations could cause cycles.

[David Osumi-Sutherland]

Hi Nomi,

Thanks for tracing this and implementing the check. No idea how the error crept in to the ontology. Guess it could be a hand edit gone wrong. I've also added a local check for this to try to prevent this happening in future.

Thanks again,

David

David Osumi-Sutherland, PhD
Ontologist / Curator
Virtual Fly Brain / FlyBase
Department of Genetics
University of Cambridge
Downing Street
Cambridge, CB2 3EH
UK
+44 (0)1223 333 963