The Broken Code with 100% Coverage

[Serkan Camurcuoglu]

Hi all,

I've written a post discussing if it's possible to achieve and sustain code correctness with testing:

https://medium.com/@serkanc/the-broken-code-with-100-coverage-796b23fd2699

I'd like to hear your comments if the topic resonates with you.

Regards,

Serkan

[Tim Wood]

Hi Serkan,

I think that there are many ways to define the coverage of tests. I
agree with your observation that many tests may be required to show
program correctness. In order for tests to show that code is bug free,
they would probably have to execute every possible path through the
program. The number of paths through a program is exponential in the
number of branches, so exploring all paths is often infeasible. For
many programs even exploring small fractions of the paths may be very
computationally expensive.

A simple example:

int myFunction(int x)
{
int result = 0;
if(x % 2 == 0)
{
// A (even)
result++;
}
else
{
// B (odd)
result--;
}

if(x % 3 == 0)
{
// C (divisible by 3)
result++;
}
else
{
// D (indivisible by 3)
result--;
}

return result;
}

We can test all the branches with the tests values 2,3,7.

2 – branches A,D
3 – branches B,C
7 – branches B,D

However, even with this simple code where there is 100% branch and
statement coverage, those tests do not cover the path “A,C”. An
additional test case is required. For example the value (x == 6) would
cover “A,C”.

To further complicate the picture, some programs are
non-deterministic. For example, programs that are multi-threaded may
have a very large number of different ways that the threads could be
scheduled - further increasing the number of test cases that would be
required to show program correctness. Furthermore, some programs (e.g.
a web server) are designed not to terminate, how long should we run
our tests for?

I think that when designing objects it is interesting to think about
modularity, encapsulation and compositionality. Can we design objects
such that when they are composed they are likely to retain whatever
nice properties they have - without us necessarily having to test all
the compositions?

Cheers,

Tim

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Growing Object-Oriented Software" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to growing-object-oriente...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Raoul Duke]

Let alone being able to define what "correct" is along all the code paths.

[Steve Freeman]

Sadly, no universal solutions provided here.

First, testing a system of any size is about a degree of (justified) confidence. Good coverage raises confidence but does not provide a guarantee. That said, I've worked on too many systems that are out of control because they have nowhere near enough regression testing.

What you might be missing from this is the idea of finding a good representation for the tests in your domain. Done well, the representation should highlight missing cases and possibly trigger new concepts. Ward's FIT had elements of this. This is hard work which should not be delegated to "test automation engineers" in some English-like tool. Unfortunately, this message got lost many years ago in the trend to adopt tooling.

S

[Selim Öber]

A really nice article warning against over dependence on tests.

I would differentiate between "fearlessly modify our code in any way we like, because we have tests" and adding new functionality though.

The difference is not only a technical one but of scope, granularity and involved parties as well. 

When I'm in fearless mode, I presume I don't change/break the contract (or the interface). If I make that modification, none of my clients (or downstream services, or consumers of the API) will notice the change. These are mostly smaller or "technical" tweaks/refactoring.

In prudent mode, as the one on the article implementing refund should have been, we are no longer talking about a modification in the code but a change in the business. This is no longer the responsibility of one person or role but more of a cross functional group of people, like domain experts, BAs, QAs and Devs. So following refund scenario, I would expect it to be implemented as a sum of many small parts and that reporting case to be caught long before development. By long, I don't mean waterfall long but more of an iteration long.

--

Selim Öber

> To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriented-software+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Growing Object-Oriented Software" group.

To unsubscribe from this group and stop receiving emails from it, send an email to growing-object-oriented-software+unsubscribe@googlegroups.com.

[Pablo Albizu Balerdi]

Hi!

A very good and direct example about this topic (by @programania):

https://programania.github.io/testing/100_percent_test_coverage_mirage.html

As usual, I think than Ken Beck's vision is very eye-opening: https://twitter.com/KentBeck/status/812703192437981184

¡Thanks for sharing your ideas!