pytest coverage testing is a wow

[Edward K. Ream]

pytest --cov has revolutionized how I work.

Rev 41ac1b in the fstrings branch now contains unit tests that cover 100% of the TOG, TOT, Fstringify and Orange branches.

What's revolutionary is the interaction between coverage and the code being tested. Over and over again I realized that uncovered code wasn't needed. I just removed it! This is completely safe, because...

If errors are discovered later one or more unit tests will be added, along with new code. Running all unit tests will ensure that all code, both old and new, work as expected. Furthermore, running all unit tests will reveal places where code becomes dead, as far as the unit tests are concerned. But removing dead code is complete safe, because...

In TDD we assume the unit tests are the last word. If they pass then by definition all is well.

Summary

Up until now, I have always hoped that unit tests were strong enough. Coverage testing eliminates all reliance on hope.

Once unit tests cover 100% of the code, any changes to the code should be valid, provided that:

a) All unit tests still pass and

b) All code remains covered.

The last two days have been some of the most productive of my life. All worry has been removed. Covering 100% of the code in unit tests is a stronger test than I ever dreamed possible. The freedom this creates is comparable to the difference in safety between C and python.

Edward

[Edward K. Ream]

On Thursday, January 16, 2020 at 3:51:10 PM UTC-5, Edward K. Ream wrote:

pytest --cov has revolutionized how I work.

...

Once unit tests cover 100% of the code, any changes to the code should be valid, provided that:

a) All unit tests still pass and

b) All code remains covered.

When I began this reply, I thought I was going to describe a "cute trick" for testing the Orange class.  However, I soon realized that one unit test illustrates an essential technique for getting 100% coverage while unit tests are failing. As I write this, I have just realized what a big deal this is.

The test_one_line_pet_peeves function contains a table of snippets. Modulo some housekeeping, it is:

for contents in table: # table not shown. Some tests in the table fail.
    contents, tokens, tree = self.make_data(contents, tag)
    expected = self.blacken(contents)
    results = self.beautify(contents, tag, tokens, tree)
    if results != expected:
        fails += 1
assert fails == 0, fails

All the "one-line tests" run, even if some fail.

It's also convenient to "pretend" that the test passes even when it doesn't. That way the -x (fail fast) option doesn't bother us.  So, for now, the last line is:

assert fails == 13, fails

As I fix the unit test failures I'll decrease the error count.

Summary

When I started this reply, I thought I was going to describe just a cute trick, namely using the call to self.blacken to get the expected results, that is, the results that black itself delivers. Pretending that the test passes is another cute trick.

Those are indeed useful tricks, but the essential trick is to run all the unit tests even while some are failing. That's the only way to ensure 100% coverage. That's a big, big deal, because without 100% coverage we have no way of knowing whether code is dead or not!

Surely you can sense my excitement. This work flow is way, way, way better than I ever thought possible.

Edward

[Brian Theado]

On Thu, Jan 16, 2020 at 4:20 PM Edward K. Ream <edre...@gmail.com> wrote:

[...]

The test_one_line_pet_peeves function contains a table of snippets. Modulo some housekeeping, it is:

for contents in table: # table not shown. Some tests in the table fail.
    contents, tokens, tree = self.make_data(contents, tag)
    expected = self.blacken(contents)
    results = self.beautify(contents, tag, tokens, tree)
    if results != expected:
        fails += 1
assert fails == 0, fails

All the "one-line tests" run, even if some fail.

 

It's also convenient to "pretend" that the test passes even when it doesn't. That way the -x (fail fast) option doesn't bother us.  So, for now, the last line is:

assert fails == 13, fails

As I fix the unit test failures I'll decrease the error count.

It is possible to let pytest do the bookkeeping for you by using a pytest decorator to parameterize the test. See https://docs.pytest.org/en/latest/parametrize.html.

Also, the @pytest.mark.xfail decorator (https://docs.pytest.org/en/latest/skipping.html#xfail-mark-test-functions-as-expected-to-fail) is useful for "a bug not yet fixed" which sounds like what you are describing.

The parameterize decorator supports marking the input arguments with xfail independent of each other.

[Edward K. Ream]

On Thu, Jan 16, 2020 at 9:56 PM Brian Theado <brian....@gmail.com> wrote:

> ...the @pytest.mark.xfail decorator (https://docs.pytest.org/en/latest/skipping.html#xfail-mark-test-functions-as-expected-to-fail) is useful for "a bug not yet fixed" which sounds like what you are describing

Thanks for this. It would have been useful.

Here are two replies to my original posts.

>> All the "one-line tests" run, even if some fail.

This is the key condition, no matter how it is accomplished. It's a matter of attitude/technique.

>> It's also convenient to "pretend" that the test passes even when it doesn't. That way the -x (fail fast) option doesn't bother us.

The easy way is not to use -x. Again, it's a matter of attitude/technique. When using pytest --cov we always want to run all the tests.

Edward