SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

Yesterday will surely rank as one of the most important days in Leo's history.

tl;dr: It will be straightforward to provide complete coverage tests for most of Leo!

Like all momentous Ahas, it has suddenly become difficult to remember what the world looked like before the Aha. I am writing this post in the middle of the night, attempting to recreate the Aha, explain the way forward, and get back to sleep :-)

A slow-motion Aha

Yesterday I thought I was finally done with work on Leo. At last, I could start studying web technologies. Indeed, yesterday I completed the "Hello World" example for VS Code plugins. But then fate intervened.

My subconscious nagged me about completing the elimination of c.onBodyChanged. This is #1733. Yesterday I reopened this item, removed the "Won'tDo" label, and reassigned it to Leo 6.4. I did this because I saw that after simplifying c.onBodyChanged it would be easy to replace c.onBodyChanged with a small amount of code, given in #1733.

I eliminated several calls to c.onBodyChanged using this strategy.  I ran all unit tests, and they all passed. So far, seemingly so good.

But now my subconscious started screaming at me: "Why are you doing this work? You have no idea whether your changes have been covered by unit tests. Admit it. You are flying blind."

According to github, I created #1758 (Cover most of Leo with unit tests) two days ago. I am quite sure that when I created this item I did not have any idea how to do it! But presumably, this item primed my mental pump. That's probably why my subconscious was screaming at me yesterday.

Yesterday afternoon Rebecca and I took a long walk in some lovely woods overlooking the University of Wisconsin. After the walk, I saw the start of the way forward.

Strangely, I don't remember the exact moment. I remember discussing unit testing with Rebecca after the walk. Before dinner, I wrote a script to convert unit tests for edit commands in unitTest.leo to "traditional" unit tests.

It was only after discussing that script with Rebecca at dinner that I started to realize what I had just done! As I talked, I got more and more excited. The building excitement was similar to the "hand-waving Aha" about synchronizing tokens in leoAst.py.

The first conversion script

The second comment of #1758 shows the first conversion script. The setup is:

- Copy @file activeUnitTests.txt-->Organized by file-->leoEditCommands
  from unitTest.leo to the top-level node of leoPy.leo.
- Create a new top-level node called 'new-tests'.
- Run the script.

The second comment also shows the output of the script, which is a "traditional" unit test.

Important: The script creates a list of Leo nodes. I'll copy these nodes to leoEditCommands.py. But like all other nodes in @file trees, the nodes create flat text in the external file. This means...

Aha 1: pytest does not have to know about Leo outlines!

Support classes

A new file, leo/core/leoTest2.py, will define the classes and helpers needed to make the new tests work. I'll adjust the conversion script to bring the generated nodes into "alignment" with the code in leoTest.py.

Aha 2: The kinds of support that leoTest2.py is much simpler than that contained in leoTest.py!

leoTest.py creates "flat" unit tests for python's unittest module. It's messy. In contrast, leoTest2.py just needs to define some base classes and simple support helpers.

Aha 3: here is no way to run coverage tests from unitTest.leo. unitTest.leo must go.

None of these three Ahas were clear to me until my dinner with my muse :-)

Summary

Running coverage tests is the crucial first step. Coverage failures will guide the testing process.

Scripts will convert unit tests in unitTest.leo to corresponding nodes containing traditional unit tests. I'll copy the generated nodes to the corresponding source files. The actual unit tests will be flat text in the external source file.

A new file, leoTest2.py, will provide support for the new testing framework. leoTest2.py will be much simpler than leoTest.py because it doesn't need to convert outlines to tests.

unitTest.leo must go. It was a useful dead end.

The ekr-undo9 branch contains preliminary work on the new testing framework.

I had expected that my sabbatical would be refreshing. I did not expect a huge breakthrough in Leo's world.

Edward

[Edward K. Ream]

On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:

tl;dr: It will be straightforward to provide complete coverage tests for most of Leo!

The various Ahas have straightforward consequences:

Leo will continue to support @test and @suite nodes. Eventually, leoTest.py will contain only the minimum code to support generic @test and @suite nodes.

Considerable code will migrate from leoTest.py to leoTest2.py. leoTest2.py will also contain new code and classes. As before, most of Leo's unit tests must be aware of Leo's DOM and API. To make that happen, leoTest2.py will define test classes that define off-screen test commanders. leoTest.py already does this. The corresponding code in leoTest2.py should be substantially simpler because the code will not be running inside Leo.

Oh joy. The distinction between running unit tests internally or externally will disappear.

leoTest2.py must provide a way of running all unit tests. It will likely do that by looking for source files with a top-level "pytest_main" method.

20+ years of hindsight should improve leoTest2.py. For example, it's now clear that most unit tests can and should be run with a "null" (string-based) gui. leoTest2.py must also support desperately needed unit tests of the Qt gui. I'll be experimenting with running tests in an "off-screen" Qt gui.

Let me emphasize several points:

1. The conversion script is dead simple, both in concept and in code. The script extracts data from a tree of nodes copied from unitTest.py, and uses that data as kwargs to a "run_test" method of a test class. Neither the run_test method nor the test class exists at present, but that's a nit :-) I'll adapt that script to convert other @test nodes in unitTest.py. Other @test nodes can just be copied to their own unit tests.

Aside: The new unit tests will not have automatic access to c and g and p. However, that won't change much, because the various setup and teardown methods can compensate. Surely, it's no big deal.

2. The sabbatical will continue. The main focus of my sabbatical remains learning new web technologies. But now a secondary focus will be to revolutionize Leo's unit tests.

In the next few weeks, I will create leoTest2.py and create new unit tests covering most of Leo's commands. PR 1759 tells which files were changed. To that list I'll add files changed in the ekr-undo9 branch.

Other work can wait. I am not going to commit to covering all of Leo's source code immediately. Some unit tests can wait. Otoh, full coverage testing of Leo's code base will be a defining feature of Leo 6.4.

Summary

No new significant invention is needed to make full coverage testing of Leo's code base a reality.

The sabbatical will continue, now with twin objectives.

Edward

[Edward K. Ream]

On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:

> The ekr-undo9 branch contains preliminary work on the new testing framework.

Not sure that's true. Anyway, work on this topic will be done in the ekr-unit-test branch. PR 1762 records the work.

After some thought, I've decided to base ekr-unit-test on devel, as usual. The goals are to minimize diffs, and to make the unit test work distinct from the undo work. When both branches are ready, I'll merge ekr-undo9 into devel first, merge devel into ekr-unit-test, and run all the new unit tests.

Edward

[Brian Theado]

Having measurable code coverage of the automated tests will be useful.

The coverage.py library is not at all tied to pytest. About a year ago I tried running its standalone version against Leo's existing tests.

However, some of the coverage results were not making sense. Possibly the way leo's unit test framework loads the code was causing confusion to coverage.py. I didn't do any investigation to see if that is a reasonable guess.

All this to say that your planned approach of extracting the tests into standard pytests is probably the better path to reliable code coverage measurements.

Have you considered using pytest features such as fixtures and parameterized marks?

I tend to favor composition over inheritance and functions over classes. The fixture feature helps with this preference.

I think your preference is the opposite, but I thought I'd ask.

--
You received this message because you are subscribed to the Google Groups "leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/7debef12-60e1-4463-9dda-8bd6a1ab7dc4n%40googlegroups.com.

[tbp1...@gmail.com]

I'm glad to see you taking this approach.  I think that different development tasks should be done in separate branches.  In the past I've not always been careful about this, and by now I think I may finally have learned the lesson.

[Edward K. Ream]

On Sun, Nov 29, 2020 at 8:49 PM Brian Theado <brian....@gmail.com> wrote:

Having measurable code coverage of the automated tests will be useful.

The coverage.py library is not at all tied to pytest.

I had forgotten that.

The big picture: Leo's old unit-testing framework, based on @test nodes in unitTest.leo, is inflexible. To add new features, complex code must be added to leoTest.py.

In contrast, leoTest2.py contains only a few simple classes, subclasses of unittest.TestCase. I am about to simplify the code further, by adding top-level helper functions in leoTest2.py, for use by various setUp methods. This may reduce or eliminate the need for base classes.

To emphasize: leoTest2.py has easy access to all of the features of python's unittest module, and by extension, all of the features of pytest, nose, whatever. This is the real breakthrough.

About a year ago I tried running its standalone version against Leo's existing tests.

However, some of the coverage results were not making sense. Possibly the way leo's unit test framework loads the code was causing confusion to coverage.py. I didn't do any investigation to see if that is a reasonable guess.

All such problems should go away, possibly to be replaced by new complications :-)

For example, there is the question of how to init g.app only once for a suite of tests. Googling this yields several useful leads involving per-class and per-module fixtures. I'm still wrapping my head around this.

All this to say that your planned approach of extracting the tests into standard pytests is probably the better path to reliable code coverage measurements.

I agree. Yesterday's work completely proves the concept. I'll say more in a new post.

Have you considered using pytest features such as fixtures and parameterized marks?

Not yet :-) My present plan is to create test functions with a single test class, rather than creating separate test classes. This seems like the simplest thing that could possibly work.

While I was debugging the new code I switched back and forth between running the test class with unittest and with pytest. For debugging, unittest seems a bit better. So I don't necessarily want to tie the code to pytest features. Unless they add something essential.

I tend to favor composition over inheritance and functions over classes. The fixture feature helps with this preference.

In the present environment, the difference seems small. However, this comment nudged me towards creating top-level helper functions in leoTest2.py. That seems like a step forward.

I think your preference is the opposite, but I thought I'd ask.

I don't have a strong preference at present.

Many thanks for all your comments. They have been helpful.

Edward

[Edward K. Ream]

On Sun, Nov 29, 2020 at 9:10 PM tbp1...@gmail.com <tbp1...@gmail.com> wrote:

I'm glad to see you taking this approach.  I think that different development tasks should be done in separate branches.  In the past I've not always been careful about this, and by now I think I may finally have learned the lesson.

Thanks for this comment.

My present plan regarding the ekr-undo and ekr-unit-test branches is as follows:

1. (Done) Document both branches in Leo's branch info page.

2. Merge ekr-unit-test into devel in stages.

    After each merge, merge devel back into ekr-undo.

3. Merge ekr-undo into devel later, when Leo has better unit tests.

Summary: The lack of complete coverage tests is not an immediate crisis :-)

Edward