Leo's new unit testing world

[Edward K. Ream]

Using "proper" (text-based) unit tests is a revolution. I am embarrassed by all the cruft in unitTest.leo and leoTest.py. What was I thinking?

unitTest.leo gets in the way of unit testing:

- The test runners in leoTest.py are inferior to the test runners in unittest and pytest.

- @test nodes are inferior to test functions and classes.

- Tests in unitTest.py can interfere with each other in subtle ways.

The unit tests in leo/unittests/commands/editCommands.py show how easy it is to use unit tests to test Leo itself:

The create_app function (in leoTest2.py) is the only needed support. It creates a fully functional Leo app outside of Leo, binding g.app to the created app. Perhaps this is what I missed so many years ago.

The typical setUpClass method just calls create_app. Thereafter, setUp methods can create a commander as follows:

self.c = leoCommands.Commands(fileName=None, gui=g.app.gui)

And away we go. Tests have full access to g.app, g.app.gui, and a fully functional commander, c.

New directions

The new testing world suggests three major improvements:

1. Create tests for the console gui. There is no need to run these unit tests from plugins/cursesGui2.py!

2. Create tests for Qt gui. There is no need to run these tests from unitTest.leo!

3. Remove g.app.unitTesting and g.unitTesting. There is no need to disable traces, etc. when running tests outside of Leo. Much better to make tests dependent on --trace values.

Acknowledgment

I spent considerable time last week on more complicated schemes for the new unit tests. That complexity eventually collapsed. Only create_app remains. That collapse would likely not have happened were it not for Brian's comment:

"I tend to favor composition over inheritance and functions over classes. The fixture feature helps with this preference."

Thanks, Brian.

Now I see that @test x is precisely equivalent to def test_x. Oh my, I have wasted so much time on feeble substitutes for features of python's unittest module.

Summary

Nothing is gained by unitTest.leo, @test, and leoTest.py. The new testing world is better all ends up.

When running tests from the command line, the create_app function in leoTest2.py gives unit tests full access to all of Leo's source code.

Edward

[Edward K. Ream]

On Friday, December 4, 2020 at 6:39:59 AM UTC-6 Edward K. Ream wrote:

> The new testing world suggests three major improvements:

#1758 (replace unitTest.leo with tests in leo/unittests) lists these items. They will be done after the sabbatical ends.

I'll soon add new unit tests for #1757 (vim bindings). This is a super tricky issue, one that cries out for better tests. At present, the code in the ekr-vim branch does not handle the default binding for <return> properly when the focus is in a headline.

Edward

[Edward K. Ream]

On Friday, December 4, 2020 at 6:39:59 AM UTC-6 Edward K. Ream wrote:

> I am embarrassed by all the cruft in unitTest.leo and leoTest.py. What was I thinking?

I probably thought that unit tests had to be run within Leo in order to test Leo. But that's wrong, as the create_app function shows.

Big sigh. I have wasted a lot of time on unitTest.leo and leoTest.py.

Edward

[Brian Theado]

Edward,

On Fri, Dec 4, 2020 at 7:40 AM Edward K. Ream <edre...@gmail.com> wrote:

I spent considerable time last week on more complicated schemes for the new unit tests. That complexity eventually collapsed. Only create_app remains. That collapse would likely not have happened were it not for Brian's comment:

"I tend to favor composition over inheritance and functions over classes. The fixture feature helps with this preference."

Thanks, Brian.

I'm glad my comment was helpful.

 

Now I see that @test x is precisely equivalent to def test_x. Oh my, I have wasted so much time on feeble substitutes for features of python's unittest module.\

Oh well.  Don't be too hard on yourself :-).

I think the code you have for checking actual vs. expected results is another opportunity for simplification.

When I change one of the test_editCommands.py expected results in order to force it to fail and then run it with pytest, I see output like this:

        if s1 != s2:  # pragma: no cover

            print('mismatch in body')

            g.printObj(g.splitLines(s2), tag='expected')

            g.printObj(g.splitLines(s1), tag='got')

            print('parent_p.b', repr(self.parent_p.b))

>           assert False

E           AssertionError: assert False

leo/unittests/commands/test_editCommands.py:54: AssertionError

This is pytest being helpful and showing exactly where in the code the failure came from.  Following that output, this is also displayed:

----------------------------------------------------------------------------- Captured stdout call ------------------------------------------------------------------------------

mismatch in body

expected:

[

    'first line\n',

    '    line 1\n',

    '        line a\n',

    '            line b\n',

    '    line cXXXXXXx\n',

    'last line\n'

]

got:

[

    'first line\n',

    '    line 1\n',

    '        line a\n',

    '            line b\n',

    '    line c\n',

    'last line\n'

]

parent_p.b ''

This is your code being helpful by showing more details about how the actual was different from the expected.

However pytest automatically tries to be similarly helpful if you are more specific with your assertions.  IOW if you change the code from this:

        if s1 != s2:  # pragma: no cover
            print('mismatch in body')
            g.printObj(g.splitLines(s2), tag='expected')
            g.printObj(g.splitLines(s1), tag='got')
            print('parent_p.b', repr(self.parent_p.b))
           assert False

to this:

       assert s1 == s2

then the pytest output looks like this:

        s1 = self.tempNode.b

        s2 = self.after_p.b

>       assert s1 == s2

E       AssertionError: assert 'first line\n...\nlast line\n' == 'first line\n...\nlast line\n'

E         Skipping 56 identical leading characters in diff, use -v to show

E         -     line c

E         +     line cXXXXXXx

E         ?           +++++++

E           last line

IMO, this is as good if not better than your output. The 'mismatch in body' message is missing from this, but maybe renaming s1 and s2 to body1 and body2 would be self-documenting and serve as a suitable replacement.

I get the feeling you are trying to avoid being dependent on pytest features. But this approach will still be able to run using only unittest, just that the output won't be as nice.

Brian

[Edward K. Ream]

On Sun, Dec 6, 2020 at 2:17 PM Brian Theado <brian....@gmail.com> wrote:

> I think the code you have for checking actual vs. expected results is another opportunity for simplification.

...

> [for assert s1 == s2 pytest shows]...

        s1 = self.tempNode.b

        s2 = self.after_p.b

>       assert s1 == s2

E       AssertionError: assert 'first line\n...\nlast line\n' == 'first line\n...\nlast line\n'

E         Skipping 56 identical leading characters in diff, use -v to show

E         -     line c

E         +     line cXXXXXXx

E         ?           +++++++

E           last line

> IMO, this is as good if not better than your output.

Thanks for this. I seem to remember that unittest has similar capabilities. I went looking for them recently but didn't see them. There may also be some python text utils somewhere that zero in on text differences.

Anyway, when I next turn my attention to unit testing I'll be sure to remember your comment.

Edward