Second try: stupendous Aha re unit tests
Edward K. Ream
http://groups.google.com/group/leo-and-pylint/browse_thread/thread/3cf874d0b41eefab
I should have remembered that nobody reads long posts. So here is the
Aha in a nutshell:
Unit tests are not just for testing! They are *the* master tool for
programming, design, testing, refactoring, studying code, or *anything
else*.
Think of a unit test as the root of a tree. The tree represents any
task (including designs).
The unit test *formalizes*, *automates* and *protects* the task or
design.
Expanding the notion of unit tests this way is a *stupendous* Aha. It
has *totally altered* how I approach my work.
For details, read the long post. Print it out, including the lengthy
"reply". Study it. Respond. Please.
Edward
P.S. I expect three possible responses to this post, and the longer
post:
1. Yeah, I knew that. Welcome to the club of highly effective
programmers.
2. Wow! I didn't know that. This is going to change my life.
3. Huh? I have no idea what you are talking about. Your writing
sucks. Can't you do a better job of explaining your ideas?
EKR
Terry Brown
"Edward K. Ream" <edre...@gmail.com> wrote:
> Unit tests are not just for testing!
[snip]
> P.S. I expect three possible responses to this post, and the longer
> post:
>
> 1. Yeah, I knew that. Welcome to the club of highly effective
> programmers.
>
> 2. Wow! I didn't know that. This is going to change my life.
>
> 3. Huh? I have no idea what you are talking about. Your writing
> sucks. Can't you do a better job of explaining your ideas?
4. I read your post, and thought, again, I really need to try these unit test things.
Random thoughts.
I think you main point was about test driven design, but you spent a lot of time emphasizing that unit tests are able to pay permanent attention to possible issues, i.e. that they never forget to check things. Which obscured the test driven design part a bit, even though permanent attention is also a valuable part of unit tests.
Writing unit tests to make sure that a bug which was reported in the wild does not recur is useful, but not a great advertisement for unit tests, seeing they didn't prevent the bug hitting the user. You post wasn't about that, but a lot of the other unit test mentions here are.
So basically, you want code that, given X, produces Y. You write a test which would test such a piece of code, if it existed, and then write the code. The unit test approach has provided a probably valuable chance to think about / clarify the design, and it continues to provide value by catching anything that breaks the designed functionality.
I think what holds me back from using unit tests more is the difficulty in writing the tests themselves (not the testing framework). A lot of code I write has some sort of server / client (web or sql server) aspect which makes constructing a pretend X fiddly. Not that it can't be done, just that it takes time, and is complex enough to be bug prone itself. And my work timeframes are not conducive to giving things the time and attention they need.
When I write something with simpler inputs and outputs, like a parser for all the conceivable variations on postgres://foo:p...@bar.com:5432/dbname.schemaname.tablename then chances are I would write a simple test.
And I'm not sure they help with everything. I spent an hour last night trying to work out why my generic code for comparing two databases for differences (after dual entry for QA) wasn't working. This is an example of a complex X. In the end the problem was forgetting how the record location reporting function worked, and so not setting some parameters it needed. I.e. it was working perfectly, I was just using it wrong.
I expect some variation on the following response to this post :-) :
You should break the problem down into smaller pieces and if you understood it properly it would be easier to write unit tests and they'd be valuable.
This is probably true.
Cheers -Terry
zpcspm
> Writing unit tests to make sure that a bug which was reported in the wild does not recur is useful, but not a great advertisement for unit tests, seeing they didn't prevent the bug hitting the user. You post wasn't about that, but a lot of the other unit test mentions here are.
To minimize the chance that this happens, having unit tests is not
enough. You must have tests that cover all the code. Unit tests in
combination with code coverage analysis are much more powerful than
unit tests alone. Personally I was pleasantly surprised to learn about
coverage.py and its ability to work as nose plugin. This was a very
considerable addition to my python toolchain. Previous ones were clone
digger, pylint and leo (but of course!).
> I think what holds me back from using unit tests more is the difficulty in writing the tests themselves (not the testing framework). A lot of code I write has some sort of server / client (web or sql server) aspect which makes constructing a pretend X fiddly. Not that it can't be done, just that it takes time, and is complex enough to be bug prone itself. And my work timeframes are not conducive to giving things the time and attention they need.
Tests involving interaction with a web/sql server, any networking I/O
and such, are not unit tests. Period. I was making the same mistake
for quite a few years. The AHA happened when I've read the book
"Working Effectively with Legacy Code" by Michael Feathers. I suggest
you to get that book and read the definition of unit tests: unit tests
are small, they test small code snippets and they complete quickly. If
this is not true - you can't call them unit tests.
> You should break the problem down into smaller pieces and if you understood it properly it would be easier to write unit tests and they'd be valuable.
> This is probably true.
This is similar to what I've just said above, so you are on the right
path.
val bykovsky
To me your post is very interesting and needs
time to think it through and respond (i enjoyed
reading it and digesting each point/thought).
It is certainly thought-provocative. To me,
it is about a fundamental mechanism of programming.
Just a brief preliminary comment for now (later
with details, hopefully). Many people noticed a mystery
of programming. I tried to understand programming
from control science viewpoint, and the idea was that
a building a program is a inverse problem, and what
a programmer does is to use the "use cases", "unit tests"
and similar constraints to reconstruct step-by-step
the program using the mismatches as a feedback to update/correct
the program. Each constraint typically has input/output or similar
format (e.g., context/data).
Imagine a 'programmable' (that is, flexible, reconfigurable)
material. Each constraint is in fact a control which drives an
iterative process of reconfiguring the material so that the
constraint is matched. Sure, the material can be pre-configured
('the art of programming') to make its programming "more efficient"
(vs. doing it from scratch).
So, looking from this angle, you came to the heart of programming.
Unit tests is a reference/control info to incrementally build-up
a program - 'configure' a programmable material. To me, this is
how animals (and some humans) 'effortlessly' program their behavior
under control of 'environmental challenges'.
Sure, this line of thinking has a long history, but in the
programming world i think this is relatively fresh and exciting
development, and your focus on tools for code exploration and
understanding seems to me extremely valuable. Actually, this
'probing' technique is applicable to exploration of the whole class of
'complex systems'.
Does this 'angle' make any sense?
My best, Val
Terry Brown
zpcspm <zpc...@gmail.com> wrote:
> Tests involving interaction with a web/sql server, any networking I/O
> and such, are not unit tests. Period. I was making the same mistake
> for quite a few years. The AHA happened when I've read the book
> "Working Effectively with Legacy Code" by Michael Feathers. I suggest
> you to get that book and read the definition of unit tests: unit tests
> are small, they test small code snippets and they complete quickly. If
> this is not true - you can't call them unit tests.
Ok, fine, so the cases for which I find it particularly difficult to write unit tests aren't appropriate for unit testing anyway. That makes me feel better I guess.
Perhaps I should be writing unit tests for small parts of the processing of the complex entity after it's been acquired from the client or server or whatever.
Cheers -Terry
Edward K. Ream
> I think you main point was about test driven design, but you spent a lot of time emphasizing that unit tests are able to pay permanent attention to possible issues, i.e. that they never forget to check things. Which obscured the test driven design part a bit, even though permanent attention is also a valuable part of unit tests.
My main point is that unit tests are general purpose tools.
What makes this work is that it's easy to run one or all unit tests
automatically.
> Writing unit tests to make sure that a bug which was reported in the wild does not recur is useful, but not a great advertisement for unit tests, seeing they didn't prevent the bug hitting the user. You post wasn't about that, but a lot of the other unit test mentions here are.
Correct. My post was about using unit tests in new ways.
> So basically, you want code that, given X, produces Y. You write a test which would test such a piece of code, if it existed, and then write the code.
That's unit testing. The Aha is that you can replace your description
by:
You want a tool that given any task or design X, verifies (or helps
with) some property Y of X.
> The unit test approach has provided a probably valuable chance to think about / clarify the design, and it continues to provide value by catching anything that breaks the designed functionality.
True.
> I think what holds me back from using unit tests more is the difficulty in writing the tests themselves (not the testing framework). A lot of code I write has some sort of server / client (web or sql server) aspect which makes constructing a pretend X fiddly. Not that it can't be done, just that it takes time, and is complex enough to be bug prone itself. And my work timeframes are not conducive to giving things the time and attention they need.
That's true, without the Aha. With the Aha, the unit test becomes the
*easy* way of executing your code. The "test" doesn't need to do
testing: it's simply there to help you in whatever way you want. So
start by having the test help you do what you want to do quickly,
namely run your code :-)
@test
<set up and run your code>
Do you see? Create a Leo file, put this node in the file, then do
Alt-6 to run all @test nodes externally. There is only one @test
node, so Alt-6 "just works". Because the tests are run externally,
they use the latest version of your code: no need to reload Leo. It's
a *big* step forward in speed.
> When I write something with simpler inputs and outputs, like a parser for all the conceivable variations on postgres://foo:p...@bar.com:5432/dbname.schemaname.tablename then chances are I would write a simple test.
Write the test or not as you please. The @test node above will save
you time.
> And I'm not sure they help with everything. I spent an hour last night trying to work out why my generic code for comparing two databases for differences (after dual entry for QA) wasn't working. This is an example of a complex X. In the end the problem was forgetting how the record location reporting function worked, and so not setting some parameters it needed. I.e. it was working perfectly, I was just using it wrong.
So write an @test node that will sniff out the data you want to see.
For example, you might want to put a "sniffing" top-level var in the
code. The @test node will set this var, something like this:
@test
import mysteryCode
mysteryCode.sniffing = True
mysteryCode.run()
This is only a general idea. Once you see @test nodes as general
tools, not necessarily related to testing, but useful for *any* task
you are doing, you have gotten the Aha and its power.
Edward
Edward K. Ream
On Feb 22, 10:21 am, zpcspm <zpc...@gmail.com> wrote:
> To minimize the chance that this happens, having unit tests is not
> enough. You must have tests that cover all the code.
Just today, as the result of a post in Planet Python, I started
looking at coverage.py. It's cool.
http://nedbatchelder.com/blog/201002/back_from_pycon_2010.html
Edward
Edward K. Ream
> To me your post is very interesting and needs
> time to think it through and respond (i enjoyed
> reading it and digesting each point/thought).
Thanks.
> It is certainly thought-provocative. To me,
> it is about a fundamental mechanism of programming.
Yes.
> So, looking from this angle, you came to the heart of programming.
> Unit tests is a reference/control info to incrementally build-up
> a program - 'configure' a programmable material.
I agree. We already have these really cool controllers. They are
called human beings :-)
Human controllers have all kinds of ideas and desires, but we aren't
real good at maintaining attention, and handling myriad details. So
we need help. Unit tests are that help. They are as flexible as
their human controllers make them. The collaboration of humans and
@test is superb: each can do what it does best.
> this 'probing' technique is applicable to exploration of the whole class of
> 'complex systems'.
Yes.
> Does this 'angle' make any sense?
It certainly does.
Edward
Edward K. Ream
> @test
> import mysteryCode
> mysteryCode.sniffing = True
> mysteryCode.run()
Actually, the headline of the @test node would be something like:
@test sniff out the mystery code
This will be obvious to those who already use @test, but it's good to
emphasize for @test newbies that the headline becomes the name of the
unit test. So the UnitTest.testRunner will report that name if
something goes wrong.
Perhaps more importantly, the @test headlines become a form of
documentation that is readily available. It's easy to see all your
@test nodes when you want, and it's easy to squirrel them away when
you don't want to see them.
Edward
Edward K. Ream
On Feb 22, 10:44 am, "Edward K. Ream" <edream...@gmail.com> wrote:
> Human controllers have all kinds of ideas and desires, but we aren't
> real good at maintaining attention, and handling myriad details. So
> we need help. Unit tests are that help. They are as flexible as
> their human controllers make them.
The Aha is that we have been using unit tests inflexibly. We have
been blinded by their name.
In fact, the essence of unit tests is that they can be run
automatically at any time. The combination of Leo's outline
organization and the UnitTest.testRunner means that we can run one,
some or all the "tests" very easily. We can run selected tests with
Alt-5, or run all tests with Alt-6.
To put it another way, the Aha is that @test doesn't define a test, it
defines one of a suite of *general-purpose scripts* that the
testRunner can run easily. Don't think "test", think "helper".
EKR
Edward K. Ream
On Feb 22, 10:35 am, "Edward K. Ream" <edream...@gmail.com> wrote:
> That's true, without the Aha. With the Aha, the unit test becomes the
> *easy* way of executing your code. The "test" doesn't need to do
> testing: it's simply there to help you in whatever way you want.
This is a good place to tell you about my new work flow for unit
tests. I typically create new tests in leoPy.leo and execute them
with Alt-6. This works well because I move tests to unitTest.leo
after I am through using them to develop code. Thus, Alt-6 executes
only new tests because that's all there are.
EKR
ne1uno
On Feb 22, 8:21 am, val bykovsky <v...@vtek.com> wrote:
> Hi Edward and All:
> To me your post is very interesting and needs
> time to think it through and respond
> Imagine a 'programmable' (that is, flexible, reconfigurable)
> material. Each constraint is in fact a control which drives an
> iterative process of reconfiguring the material so that the
> constraint is matched. Sure, the material can be pre-configured
> ('the art of programming') to make its programming "more efficient"
> (vs. doing it from scratch).
> So, looking from this angle, you came to the heart of programming.
> Unit tests is a reference/control info to incrementally build-up
> a program - 'configure' a programmable material. To me, this is
> how animals (and some humans) 'effortlessly' program their behavior
> under control of 'environmental challenges'.
> Sure, this line of thinking has a long history, but in the
> programming world i think this is relatively fresh and exciting
> development, and your focus on tools for code exploration and
> understanding seems to me extremely valuable. Actually, this
> 'probing' technique is applicable to exploration of the whole class of
> 'complex systems'.
> Does this 'angle' make any sense?
this sounds like what I call scaffolding. it prerceeds unittesting and
is the only thing that works in the initial design. stub programs,
print of info, more typical things you do while getting set to
refactor are all scaffolding, support structure removed while cleaning
uo. unittests can be a temporary or permanent part of this. subject to
bit rot like any other code, unittests too early become an unnecessary
hindrance to progress.
have you read any of edwards earlier posts? he has been unittesting
for years, I think the new workflow mentioned where valid unittests
are moved into one or more files/trees helps maintain sanity yet while
with scaffolding all you do is remove it and on to more progress.
nobody can see the exact process, it leaves little trace beyond
intermediate commits and backups. unittests are heavy scaffolding is
light and infinitely adaptable although not always repeatable in the
way unittests are. it I submit is indispensable while unittests, say
what you will, are still optional.
Matt Wilkie
...and I'm thinking, I really need to *learn* what these unit test
thingies are. Being a fledging developer, barely into kindergarten, it
all sounds wonderful, but still might as well be martian to me. :)
Where should I go to learn about unit tests, unit helpers, in a leonine way?
thanks,
--
-matt
tfer
testing is only set up to work on Leo code, that is that code set up
with unit testing can only be tested if it is run alongside Leo, not
if it is python code written with Leo but run outside of it.
Changes would have to be made to allow Leo based unit testing
functionality to be exported to standalone code.
Tom
Edward K. Ream
> Unless things have changed, it should be pointed out that Leo's unit
> testing is only set up to work on Leo code, that is that code set up
> with unit testing can only be tested if it is run alongside Leo, not
> if it is python code written with Leo but run outside of it.
It's a matter of point of view. Obviously, @test nodes only work in
Leo, but the code inside @test nodes can be any code whatever.
> Changes would have to be made to allow Leo based unit testing
> functionality to be exported to standalone code.
Such a script could be written. It would physically place the body of
the @test node inside a synthesized run() method. That is, the script
would create something like:
class munged_test_name: (UnitTest.TestCase):
def run():
<< body of @test node >>
Or something like that. The munged name of the @test node could be
used as the name of the test case/class and as the name of the
external file.
I have no particular interest in doing this, as I have no particular
interest in working outside Leo, but the script would hardly be rocket
science. All patches gratefully accepted :-)
Edward
Edward K. Ream
Just learn about @test and Leo's various unit test commands. To see a
list of those commands, do
<alt-x>run-<tab>
Edward
Edward K. Ream
> this sounds like what I call scaffolding. it precedes unittesting and
> is the only thing that works in the initial design.
The Aha is that unit tests can verify design issues as well.
I don't want to make a rigid distinction between unit tests and
scaffolding. Everything is just a (helper) script, automated by the
test runner. Of course there *are* distinctions between various kinds
of helper scripts: that's what makes them so useful.
We need not bee too concerned about what the "right" way to use
scripts are. The Aha is meant to stimulate ideas and experimentation.
Only experiments will show what is truly useful and what is not.
Edward