ENB: The new unit testing framework

[Edward K. Ream]

In this Engineering Notebook post, I'll summarize the new unit testing framework. The new code is in the ekr-unit-test branch.

I'll discuss only the most interesting features, and will be as brief as possible. See PR #2159 for the code-level details.

tl;dr: See the summary.

@test and @suite were design and coding blunders

- unitTest.leo had to be running to run Leo's unit tests.

- Each unit test required teardown code to ensure that it didn't change unitTest.leo.

- Using outline nodes as unittest data was faux clever. Plain text data is easier to understand.

- Leo's TestManager class, in leoTest.py, was a feeble imitation of python's unittest module.

Leo's new unittests in leo/unittest are separate from Leo itself

- The files in leo/unittests are completely separate from the code each file tests.

- The unittest and pytest modules "just work" on these test files.

- g.unitTesting is never true while Leo is running!

  This invariant has simplifying effects throughout Leo's code.

Example: support for TravisCI

The heart of the new version of run_travis_unit_tests.py is:

base_dir = os.path.dirname(__file__)
unittests_dir = os.path.abspath(os.path.join(

    base_dir, 'leo', 'unittests'))
suite = unittest.TestLoader().discover(unittests_dir)
runner = unittest.TextTestRunner(failfast=True, verbosity=1)
result = runner.run(suite)

Do you see? The code above does not use g, or any other part of Leo. The old version loaded Leo itself (by starting Leo's bridge), then ran tests using Leo's TestManager class.

Code-level details

Subclasses of the LeoUnitTest class (defined in leoTest2.py) contain all unit tests.

The setUpClass classmethod calls the (simple!!) create_app function (also defined in leoTest2.py) to create an instance of the LeoApp class. This instance uses a null gui and assigns a g.NullObject to g.app.db and other objects, ensuring that the LeoApp instance will be safe.

The setUp method creates a new, minimal, test outline for each separate unit test. Unit tests can easily alter this test outline!

Unit tests can do (almost) anything they like because (in effect) the LeoUnitTest class throws away each test outline at the end of each test! Unit tests are completely independent of each other!

Each unit test is responsible for setting any needed user setting.

New commands support unit and coverage testing

The new test-* commands run python's unittest module on selected parts of Leo's code. The test-all command discovers and runs all unit tests in leo/unittests.

The new cover-* commands run coverage tests on the given parts of Leo's code. There is no cover-all command.

Running test-all before pushing commits has become second nature. test-all is much faster than starting unitTest.leo!

Summary

Leo never runs during unit tests. g.unitTesting is always False while Leo runs.

The new unit tests never have to worry about damaging Leo! Unit tests can never interfere with each other.

Unit testing code for Leo is separate from Leo itself.

Python's unittest and pytest modules now work as intended. For the first time, it is possible to run coverage tests on Leo.

Leo's new test-* and cover-* commands run unit tests or coverage tests on Leo's code.

The ekr-unit-test branch eliminates unitTest.leo and replaces leoTest.py with leoTest2.py.

leoTest2.py contains only the LeoUnitTest class and the create_app function.

I'll wait until Leo 6.4 goes out the door to merge the ekr-unit-test branch into devel. In the meantime, I'll do all new unit testing in ekr-unit-test. It would be unbearable to use the old unit testing framework!

Edward

[Edward K. Ream]

On Saturday, September 11, 2021 at 4:30:28 PM UTC-5 Edward K. Ream wrote:

> In this Engineering Notebook post, I'll summarize the new unit testing framework. The new code is in the ekr-unit-test branch.

> I'll discuss only the most interesting features, and will be as brief as possible. See PR #2159 for the code-level details.

The PR changes 178 files!

As you will see from the diffs, Leo's core files have not change significantly. The changes are mostly due to the "cleanups" mentioned in the first comment of the PR. The test-all command was essential to making these changes safely.

Edward