Questions regarding the test suite

[Samuel Pastva]

Hi!

Our group has been experimenting with various ways of interfacing with SBML models from Rust and we are running into some questions regarding model validation. In particular, how things are written in the specification versus how SBML test suite implements the rules. Hopefully this is the appropriate place to ask them :)

(In the following, all questions are applicable to the Level 3 Version 2 specification)

1. Is there some official guidance regarding rule precedence? For example, consider the test case `21116-fail-01-09-sev2-l3v2.xml`. Here, an extra attribute `value` is added to `speciesReference`. This clearly violates rule 21116, which is the point of this test. However, it also violates rule 10102, which simply states that the document must not contain any attributes not defined by the SBML specification. Nevertheless, violation of rule 10102 is not reported in this test case.

2. This contrasts with my second question and is concerned with test case `10102-fail-01-30-20517-sev2-l3v2.xml`. Here, the test suite reports errors 10102 and 20517. These state that the compartment element has an attribute not allowed by the SBML Core namespace, and the model has a child not compatible with the SBML Core namespace. However, as far as my understanding of XML namespaces goes, neither of these items actually *is* in the SBML Core namespace that is being validated (i.e. level 3 version 2). These items are in the `http://www.sbml.org/sbml/version1` namespace, whereas the SBML Core namespace of this document is `http://www.sbml.org/sbml/level3/version2/core`. Shouldn't these items just be ignored? Or does the "SBML Core namespace" actually mean "any namespace of any SBML version; current, previous or future"?

3. Another "flavor" of this issue is in the test case `99107-fail-01-06-10102-sev2-l3v2.xml`. Here, error 10102 is reported due to the extra element `listOfShapes`. But this element is in the namespace `http://anything` and rule 10102 only talks about elements declared in the SBML Core namespace or package namespaces. My understanding of the rule is that it should not apply to elements in non-SBML namespaces (i.e. such elements should be allowed under this rule).

4. I guess this is again about rule precedence, but shouldn't `21105-fail-01-07-sev2-l3v2.xml` also report violation of rule `21116`? I understand that the `speciesReference` element is invalid at this position, but shouldn't 21116 still apply?

5. In the test `10311-fail-01-29-10313-sev2-l3v2.xml` (and other similar tests in this folder), the issue 10313 is declared in the filename as an "extra rule", but it is never reported in the expected output (whereas in other cases that have the extra rule, like the ones discussed above, the extra rule is part of the expected output). I understand that when 10311 is reported, 10313 is largely redundant (similar to 10102 being redundant if 21116 is reported), but then I don't understand the meaning of the "extra rule" file modifier. Is this an error that is also present in the model and should be reported in this test case, or is it an error that is technically present in the model but will not be reported due to rule precedence/other reasons?  

I understand that these are rather pedantic questions, but we are trying to understand which of these are due to the SBML core specification and which are just implementation-specific to libSBML. So thank you in advance for bearing with me and for any clarification :)

Best regards,
Sam

[Lucian Smith]

These are good questions!  And this is indeed the correct place to ask about them.  Welcome!

On Tue, Mar 5, 2024 at 8:17 AM Samuel Pastva <daem...@gmail.com> wrote:

1. Is there some official guidance regarding rule precedence? For example, consider the test case `21116-fail-01-09-sev2-l3v2.xml`. Here, an extra attribute `value` is added to `speciesReference`. This clearly violates rule 21116, which is the point of this test. However, it also violates rule 10102, which simply states that the document must not contain any attributes not defined by the SBML specification. Nevertheless, violation of rule 10102 is not reported in this test case.

I agree this can be confusing.  The backstory for these tests is that they all existed internally to test libSBML, but it seemed useful to make them more generally available to the public, particularly for groups that didn't use libsbml to perform validation prior to execution or analysis.  

But where this gets confusing is that internally, libsbml runs those tests while only checking that suite of problems, so won't find 10102 in this case.  It's perfectly fine if you do!  And it's also understandable that you thought it was a problem; we need to be clearer about this in the documentation.

 

2. This contrasts with my second question and is concerned with test case `10102-fail-01-30-20517-sev2-l3v2.xml`. Here, the test suite reports errors 10102 and 20517. These state that the compartment element has an attribute not allowed by the SBML Core namespace, and the model has a child not compatible with the SBML Core namespace. However, as far as my understanding of XML namespaces goes, neither of these items actually *is* in the SBML Core namespace that is being validated (i.e. level 3 version 2). These items are in the `http://www.sbml.org/sbml/version1` namespace, whereas the SBML Core namespace of this document is `http://www.sbml.org/sbml/level3/version2/core`. Shouldn't these items just be ignored? Or does the "SBML Core namespace" actually mean "any namespace of any SBML version; current, previous or future"?

We'll have to ask Sarah Keating about this, but I *think* what is going on is that since the 'appear' namespace is also SBML, libsbml validates it as well, and it's not valid SBML L1, so we report it.

 

3. Another "flavor" of this issue is in the test case `99107-fail-01-06-10102-sev2-l3v2.xml`. Here, error 10102 is reported due to the extra element `listOfShapes`. But this element is in the namespace `http://anything` and rule 10102 only talks about elements declared in the SBML Core namespace or package namespaces. My understanding of the rule is that it should not apply to elements in non-SBML namespaces (i.e. such elements should be allowed under this rule).

I think you're right, and this is a bug in the test suite/libsbml.

 

4. I guess this is again about rule precedence, but shouldn't `21105-fail-01-07-sev2-l3v2.xml` also report violation of rule `21116`? I understand that the `speciesReference` element is invalid at this position, but shouldn't 21116 still apply?

This does indeed go with the earlier note that this was for libsbml validation, and a 'full' validation might well be more robust.  I believe that libsbml, should it find an element in the wrong place, does not try to parse that element at all.  From the spec's perspective, you are correct that 21116 would indeed also apply.

 

5. In the test `10311-fail-01-29-10313-sev2-l3v2.xml` (and other similar tests in this folder), the issue 10313 is declared in the filename as an "extra rule", but it is never reported in the expected output (whereas in other cases that have the extra rule, like the ones discussed above, the extra rule is part of the expected output). I understand that when 10311 is reported, 10313 is largely redundant (similar to 10102 being redundant if 21116 is reported), but then I don't understand the meaning of the "extra rule" file modifier. Is this an error that is also present in the model and should be reported in this test case, or is it an error that is technically present in the model but will not be reported due to rule precedence/other reasons?  

My guess is that at one point, libsbml recorded 10313 as an 'extra error' for this model, but that later updates dropped that report, and the filename was never changed.  I think the logic in the code currently (in the tester) is that we check for extra errors, and if they're there, we check to see if the filename lists it as well.  Clearly, though, the reverse is not true, and we neglected to check to see if errors listed in the filename are actually present in the error list.

So!  Hopefully this makes it clear that these test files began life as internal-to-libsbml checks, and have been made public in the hope that they are useful, as opposed to being prescriptive about how to report errors in SBML files.  In the case of 10311-fail-01-29-10313-sev2-l3v2.xml, for example, libsbml happens to report error 10311, and fails to report 10313, but a different validator might report both, or might report just 10313!  There are not really rules about how robust a validator needs to be, other than the precedence of libsbml.  And libsbml generally reports a lot of errors, but when there are multiple errors, feels free to stop once it's found at least one.

The upshot is that the validation 'must haves' are merely:

* No invalid SBML file should crash your interpreter

* No invalid SBML file should be reported as valid.

* No validation report for an invalid SBML file should report an error that the SBML file does not have  (but this is a bit difficult to verify with the current suite, as it focuses on errors that libsbml happens to report on, and is not exhaustive).

And it is nice but not necessary for a validator to:

* Report more than one validation error per file

* Report the validation number

* Report the line number of the problem in the invalid file.

* Report what specifically caused the problem.

Another bit of trivia you might find instructive:  I actually wrote a program to go through and pull out all the test files, and do a *full* validation on all of them.  For many of them, the validation error being tested worked in context (libSBML applying a single validator to the file), but in the context of a full validation, it would find a different error and stop before it got to the error the file was ostensibly created to test.  Those, I dropped, and created the 'syntactic test suite' from the remaining files alone.  But if you find a different 'true' error in a file, that's fine; you shouldn't feel obligated to find errors in the exact same priority and precedence that libsbml does.

I understand that these are rather pedantic questions, but we are trying to understand which of these are due to the SBML core specification and which are just implementation-specific to libSBML. So thank you in advance for bearing with me and for any clarification :)

Not at all!  Pedantic questions are always welcome on a forum like this ;-)  Feel free to ask more questions or for clarification.

-Lucian

[Sarah Keating]

Hi Sam

Very excited to see somebody applying rust to sbml. I'll have a look at the one Lucian referred to me to tomorrow.

Indeed happy to discuss further. So if possible you could consider coming to our upcoming event HARMONY2024

https://co.mbine.org/events/

Sarah 

--
For questions or feedback about the sbml-discuss list,
contact sbml...@googlegroups.com
---
You received this message because you are subscribed to the Google Groups "sbml-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sbml-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sbml-discuss/CAHLmBr0ZqLkiFM-J6peyZBfvbEGzaahXhTOosvVv13DHHuxUxQ%40mail.gmail.com.

[Samuel Pastva]

Thank you very much for the answers! This is indeed very helpful!

Now I understand how we should treat the answers from the test suite, which is great. I think we'll try to mostly match your outputs for now, just for the sake of consistency. But we'll likely add some extra output to the test cases where it makes sense.

Regarding HARMONY2024, sadly I don't think I'll be able to make it. I have some other event booked that week. But maybe I can try to at least connect remotely to see what's what. I'll see if I can make it to COMBINE though. That might be more feasible schedule-wise.

TL;DR about our project: We are trying to build a library with a DOM-like API, but of course tailor made for SBML. And since we are mostly running Rust + Python in our projects, it made sense for us to make it natively in Rust (also, interfacing with C/C++ from Rust is reasonably supported, but can be rather tedious; and other native Rust libraries seem mostly not feature-complete), with the option to add Python bindings later. The main reason why we are trying to do this (aside from just having something nice in Rust) is because we believe it will eventually allow us to prototype new packages/extensions faster than in other existing libraries. Still, we will never be able to match the breadth of libSBML, so we want to stay as "compliant" with existing conventions as possible.

Thank you very much again for the replies! I'll make sure to let you know here if we find any other questions, or raise an issue in the repository if we find something that we believe could be a bug in the tests :)

Sam