Test hierarchy

[Bruno Bieth]

Hi, I've been through the Jumi documentation but I couldn't find anything about the test hierarchy.

In https://github.com/orfjackal/jumi/wiki/Frequently-Asked-Questions#why-jumi-reports-pass-2-even-though-i-ran-only-one-test I can read "This is due to Jumi's flexible execution model which allows arbitrarily nested tests". Is there a documentation on this execution model?

I guess when using JUnit on top of Jumi, this nesting ability cannot be leveraged?

Thanks

[Esko Luontola]

Bruno Bieth wrote on 1.11.2013 13:40:
> Hi, I've been through the Jumi documentation but I couldn't find
> anything about the test hierarchy.
> In
> https://github.com/orfjackal/jumi/wiki/Frequently-Asked-Questions#why-jumi-reports-pass-2-even-though-i-ran-only-one-test I
> can read "This is due to Jumi's flexible execution model which allows
> arbitrarily nested tests". Is there a documentation on this execution model?

Thanks for asking. I've now written such a documentation:
https://github.com/orfjackal/jumi/wiki/Execution-Model-and-Terminology

Please read it and tell me if there is something that you don't
understand, so I can improve the documentation.

> I guess when using JUnit on top of Jumi, this nesting ability cannot be
> leveraged?

True. In JUnit only one test method may be executed at a time, which is
a subset of what Jumi makes possible: in Jumi a test execution may
contain other tests, so that it's possible to express tests which
consist of multiple steps.

Jumi's JUnit compatibility adapter [1] uses the nesting ability only to
show in the test results that what test class contained the test method.
JUnit's Description class [2] can express only this kind of static nesting.

[1]
https://github.com/orfjackal/jumi/blob/2e3d2e25c6b53133cf16ba9c95347c7f39e6b72a/jumi-core/src/main/java/fi/jumi/core/junit/JUnitRunListenerAdapter.java#L88
[2] http://junit.sourceforge.net/javadoc/org/junit/runner/Description.html

--
Esko Luontola
www.orfjackal.net

[Bruno Bieth]

Thanks for your answer. I think it would be nice to add in the doc:

- more details about isolation: the doc gives the impression that nested tests are not isolated from their parents but maybe are from their siblings? If you intend to provide Jumi as a drop-in replacement for the JUnit runner then it should have the same semantic, especially regarding isolation.

- the use cases for this feature: again, coming from JUnit it feels hacky that the runner doesn't report the same amount of tests, so there must be a significant use case for that.

[Esko Luontola]

Bruno Bieth wrote on 4.11.2013 11:01:
> Thanks for your answer. I think it would be nice to add in the doc:
> - more details about isolation: the doc gives the impression that nested
> tests are not isolated from their parents but maybe are from their
> siblings? If you intend to provide Jumi as a drop-in replacement for the
> JUnit runner then it should have the same semantic, especially regarding
> isolation.
> - the use cases for this feature: again, coming from JUnit it feels
> hacky that the runner doesn't report the same amount of tests, so there
> must be a significant use case for that.

Thanks for the feedback. I've improved the documentation to give an
answer to those questions:

https://github.com/orfjackal/jumi/wiki/Execution-Model-and-Terminology/_compare/3e5e5a6854cfe1acc2bf15be3069fec5e72a1598...9bee8d44581f5e4f38f029e0bbfaf2e247a3de12

Changes:

- Avoid using the word "isolation", instead talk about a single or
multiple steps.
- Make it explicit that Jumi runs JUnit tests the same way as JUnit
does, because Jumi invokes JUnit.
- Give examples of use cases.

--
Esko Luontola
www.orfjackal.net