On Sunday, March 25, 2018 at 7:18:41 AM UTC-4, Rick C. Hodgin wrote:
> On Sunday, March 25, 2018 at 6:37:00 AM UTC-4, Rick C. Hodgin wrote:
> > I worked for a company in the mid-2000s that had six full-time
> > developers, and three full-time unit test writers. They wrote
> > test after test after test for the code we would write.
> >
> > I was never able to justify that degree of labor spent on writing
> > tests when you, as a developer, are supposed to test the code you
> > write as you go.
>
> I would go so far as to say that there were cases where it made for
> sloppy developers, because they knew "the unit testers will find
> any bugs."
>
> When you know the final authority stops with you, you do some extra
> double- and triple-checks to make sure. You might also ask another
> developer to look it over and see if there's anything obvious. It
> helps inter-personal relationships in understanding the hows and
> whys of what made a particular thing be designed or altered in that
> way, and it puts a second mind on the problem knowing that they may
> be thinking of things you haven't pigeonholed your own thinking in
> to while working on this isolated component.
>
> I've caught more bugs that way than I've ever caught with unit
> testing, and probably even with release-to-testers testing.
I've also looked at it from a cost factor. If a high-level developer
is paid X, and a unit tester is paid k*X where k < 1, then there is a
labor savings for the unit tester to do some of the labor.
But when you consider the time in testing, the familiarity with the
system being written and developed. the unit tester will approach the
"black box" being given the specs by the developer as to what needs
to be tested and how, which means some additional up-front labor on
behalf of the developer anyway (to document those needs), and now you
have a "blind" unit tester writing code to validate something you had
previously been there intimately with. You had the algorithms open,
the purpose of the program flow was in your mind, you were "in the
zone" in the code ... so why sacrifice that tremendous asset only to
pass on to the unit tester some ability to test your code when you
could test it right there?
And now you're back to paying X for your labor, and for your testing,
rather than X for your labor and k*X where k < 1 for the testing,
but your X labor goes up slightly to document what the tester needs,
which means you're probably back on par with what you would have cost-
wise had you done the testing yourself.
And then there's productivity per developer.
In my experience, it's far more desirable for a developer to spend a
little extra time double- and triple-checking their changes with some
last minute manual testing than it is to release something with a bug
and come back and fix it later. The developer is there in the full
mindset of the code, and is fluid during that last double- and triple-
check time. But when it's released into a build, built, and tested by
someone else (or even something else as a unit test later), the dev-
eloper must re-enter that fluid state, which takes time and increases
the risk of error because it's unlikely the developer will reach the
full fluid state without a protractive effort, and if the bug that's
being examined is non-trivial (something highly likely if a skilled
developer missed it with double- and triple-checks), then it's going
to require a notably longer period of time to fix it than it should.
In the end, I conclude that unit testing has validity in some types
of code, and in other types of code it's a waste of time, material,
and resources.
That being said, I am pushing for more unit testing in my life as I
get older because I am not able to enter that "fluid zone" as easily
or as deeply as I could when I was younger. I can at times, but not
as a matter of course as before. As a result, I would like to have
more testing to validate I haven't forgotten something I previously
fixed through my own aging-related inabilities.
-----
On additional factor, in Visual Studio 2015 there was a new native
tool introduced into the base VS package, which was called "Code
Analysis." It would go through your code and perform smart fuzzy
testing on your algorithms, and perform the "what-if" scenarios.
It would say "Okay, here's a logical test for an if block... what
if it was true, what would happen?" Then it would examine it and
report on its findings. It would then come back and say, "Okay,
now what if it wee false, what would happen?" And it would examine
that condition.
It rigorously iterates through all possible choices and looks for
a wide range of errors in your code, such as uninitialized variables,
use after free, etc.
Using this new tool, on a C/C++ code base about 30K lines long, I
found about 15 places where potential errors existed. In all but
a couple cases, they were conditions that would never exist because
there was some check higher up in the code which precluded the error
condition from ever being a possibility. But in a couple cases,
there were legitimate bugs that it found where, had that particular
scenario happened, it would've errored out or produced invalid
results.
And to this day, on that algorithm, I still have a place somewhere
in my code where it writes beyond the buffer limits on something.
I have spent days trying to find it and cannot. I've downloaded
memory tools to look for buffer overruns, ported it to other com-
pilers to see if they could find any errors. I've written my own
manual memory subsystem (which I reported on in the comp.lang.c
thread a year or two back) so I could walk the heap on each of my
allocations and releases.
I still haven't found it despite all this testing. All I've been
able to do is modify my malloc() algorithms to use my_malloc()
and allocate an extra 16 bytes before and after each requested
memory block, then return a pointer to the 16 bytes after the
start. That has solved it, but it's a hack workaround and not
a true fix.
Every now and again when I get a few spare bit, I go back in and
try to find it. It's been probably four years now and I cannot
find it.
I'm nearly convinced there isn't a problem in my code as per my
understanding of how things work, but that what I'm thinking is
a legal use of a pointer or block is actually illegal, resulting
in undefined behavior at that point. And until I'm able to learn
what it is that's causing the error, the extra buffer around the
allocated memory blocks is a sufficient work-around.
-----
To be honest, those are the areas where I could see having some
type of natural language testing abilities would be of greatest
benefit. To be able to traverse your code looking at all of the
possible code paths rigorously, reporting on things used apart
from initialization, or after releasing, or many other such tests,
coupled to the ability to get a digital map of everything done in
the memory subsystem, and to be able to provide compile-time info
regarding where a pointer should operate only, and to be able to
trap any errors of use beyond that range during this special
compilation mode which adds the checks for such things.
Those kinds of developer tools would help me more than unit test-
ing on my own algorithms as they are fairly easy to test if they're
working with a few manual cases applied during development.
At least that's been my experience.
--
Rick C. Hodgin