Test first as specification
![Nick Malik [Microsoft]'s profile photo](https://lh3.googleusercontent.com/a/default-user=s40-c)
Nick Malik [Microsoft]
items for another team to develop. What is the value of the specification
being translated to unit tests before delivery to the coding team?
Should the coding team (in India) create the tests or should we create the
tests and pass them over as a form of validation?
I would be surprised if the India team were familiar with Agile methods of
any sort, so don't expect a lot of exposure to NUnit. The concept will be
foreign.
Comments and advice?
--
--- Nick Malik [Microsoft]
MCSD, CFPS, Certified Scrummaster
http://blogs.msdn.com/nickmalik
Disclaimer: Opinions expressed in this forum are my own, and not
representative of my employer.
I do not answer questions on behalf of my employer. I'm just a
programmer helping programmers.
--
Michael Feathers
> I'm coming on to a project where the team in place is specifying framework
> items for another team to develop. What is the value of the specification
> being translated to unit tests before delivery to the coding team?
>
> Should the coding team (in India) create the tests or should we create the
> tests and pass them over as a form of validation?
>
> I would be surprised if the India team were familiar with Agile methods of
> any sort, so don't expect a lot of exposure to NUnit. The concept will be
> foreign.
>
> Comments and advice?
If it is a framework, it seems like a decent approach. For straight
application development, I'd use something like FIT/Fitnesse so that
there aren't any structural constraints, but frameworks publish
interfaces, so having tests that specify those makes sense to me. I'd
be leery about specifying internals.
Another thought, that is somewhat related: I wish that everyone writing
frameworks took the time to write UTs for code that *uses* the
framework. With many frameworks out there, it is simply impossible to
do without wrapping the framework classes. Things that get in the way
are the use of 'sealed' and non-virtual functions. Many framework
developers try to protect their code from misuse using those constructs;
what they end up doing is protecting their client's code from testing,
and that is not very useful at all.
Michael Feathers
author, Working Effectively with Legacy Code (Prentice Hall 2005)
www.objectmentor.com
Tim Haughton
"Nick Malik [Microsoft]" <nick...@hotmail.nospam.com> wrote in message
news:0qCdnaGD3JO...@comcast.com...
> I'm coming on to a project where the team in place is specifying framework
> items for another team to develop. What is the value of the specification
> being translated to unit tests before delivery to the coding team?
I'd say that specifications are at their most valuable when expressed in an
automated test format. When developer and customer are separated by
geography, timezone and language, there is the potential for ambiguity to
creep into even the most tightly worded specification. Red lights and green
lights don't lie (relativistic doppler effects not withstanding).
>
> Should the coding team (in India) create the tests or should we create the
> tests and pass them over as a form of validation?
You should definitely create them. It's always a good idea for the client to
own the interface, plus your understanding of the required solution will be
greater than theirs. If they write the tests, they will be testing against
*their* understanding.
I'd suggest formal deliveries every 2 weeks, but insist on real time
visibility of state and progress (Cruise Control, Fit etc)
> I would be surprised if the India team were familiar with Agile methods of
> any sort, so don't expect a lot of exposure to NUnit. The concept will be
> foreign.
>
They hold iteration planning games whilst clinging to the outside of
speeding trains. I can think of nothing more extreme.
--
Regards,
Tim Haughton
Agitek - Agile Technology Solutions
http://agitek.co.uk
http://blogitek.com/timhaughton
Alvin Ryder
> I'm coming on to a project where the team in place is specifying framework
> items for another team to develop. What is the value of the specification
> being translated to unit tests before delivery to the coding team?
>
Hi,
I've never recieved or provided specs as tests so I can't speak from
direct experience but I've been thinking of doing exactly the same
thing myself.
I was motivated to do it because AFAIC the greatest value of TDD is the
concrete specification it provides (the testing side is a bonus). Such
a high degree of clarity is great to have, nothing is wishy washy and
"open to interpretation".
And everything is equally important none of this "we'll worry about
those less important things later" (which invariably causes drama down
the track) ...
I'm keen to try it.
> Should the coding team (in India) create the tests or should we create the
> tests and pass them over as a form of validation?
>
> I would be surprised if the India team were familiar with Agile methods of
> any sort, so don't expect a lot of exposure to NUnit. The concept will be
> foreign.
>
In my case I was going to totally bypass the Agile/TDD learning curve
by just saying here are the "Examples" (not tests) of what I need can
you please make them work:-
void testBlah() {
Blah b = new Blah();
tester.assertTrue ("This is good", b.isGood());
tester.assertTrue ("This is 3", 3 == b.getThree())
...
}
I use my own very basic test framework so there's pretty much nothing
to learn.
> Comments and advice?
>
> --
> --- Nick Malik [Microsoft]
> MCSD, CFPS, Certified Scrummaster
> http://blogs.msdn.com/nickmalik
>
> Disclaimer: Opinions expressed in this forum are my own, and not
> representative of my employer.
> I do not answer questions on behalf of my employer. I'm just a
> programmer helping programmers.
> --
Cheers.
Tim Haughton
> I was motivated to do it because AFAIC the greatest value of TDD is the
> concrete specification it provides (the testing side is a bonus). Such
> a high degree of clarity is great to have, nothing is wishy washy and
> "open to interpretation".
I think you're talking about something different to Nick. The tests that
Nick's talking about are unit tests, which are of a much coarser granularity
to the developer tests you'd use when practising TDD. Yes, developer tests
do act as a specification, but it's a much lower level specification that's
most likely to be of interest to the developers rather than the customer.
>
> And everything is equally important none of this "we'll worry about
> those less important things later" (which invariably causes drama down
> the track) ...
There's nothing in using unit tests to specifiy a product that doesn't allow
for prioritisation of features. It would be a pretty poor apporach if it
didn't. One can still say "here is my automated test specification, and
here's the order in which I want them to pass." Providing this kind of
control for a customer is vital for confidence building.
Phlip
> I'm coming on to a project where the team in place is specifying framework
> items for another team to develop. What is the value of the specification
> being translated to unit tests before delivery to the coding team?
>
> Should the coding team (in India) create the tests or should we create the
> tests and pass them over as a form of validation?
>
> I would be surprised if the India team were familiar with Agile methods of
> any sort, so don't expect a lot of exposure to NUnit. The concept will be
> foreign.
>
> Comments and advice?
Your situation divides authority from responsibility. The specifying team
doesn't have the responsibility to prove the designs. The remote team
doesn't have the authority to improve the specifications. (And when tests
drive development, writing tests in isolation is just like drawing models in
isolation, and ordering someone else to implement the design.)
To close this loop, the local team should implement at least 3 client
applications for this framework. These applications should form part of the
test suite, and you should build two test servers, one at each site, both
triggered by the same common codebase. One should serve a testing wiki like
fitnesse, and the both should run all the acceptance tests.
And send a TDD coach to the remote team, to get them over the learning curve
and show them out easy pure TDD is.
Then share the responsibilities on both sides. Each team writes tests and
code.
--
Phlip
http://www.c2.com/cgi/wiki?ZeekLand
Alvin Ryder
> Hi Alvin,
>
> > I was motivated to do it because AFAIC the greatest value of TDD is the
> > concrete specification it provides (the testing side is a bonus). Such
> > a high degree of clarity is great to have, nothing is wishy washy and
> > "open to interpretation".
>
> I think you're talking about something different to Nick. The tests that
> Nick's talking about are unit tests, which are of a much coarser granularity
> to the developer tests you'd use when practising TDD. Yes, developer tests
> do act as a specification, but it's a much lower level specification that's
> most likely to be of interest to the developers rather than the customer.
>
I was talking about specifications in the form of coded tests. I
thought that part was the same. But in my context I wanted to specify
for a local developer not part of my team yet I want them to do bits
and pieces. Not pure XP I know but I still want the assistance. At any
rate the granularity can be very low or high level the principle is
still the same I would have thought ;-)
> >
> > And everything is equally important none of this "we'll worry about
> > those less important things later" (which invariably causes drama down
> > the track) ...
>
> There's nothing in using unit tests to specifiy a product that doesn't allow
> for prioritisation of features. It would be a pretty poor apporach if it
> didn't. One can still say "here is my automated test specification, and
> here's the order in which I want them to pass." Providing this kind of
> control for a customer is vital for confidence building.
>
I'm sure priorities would be of value in some contexts but in my case
I'd prefer an all or nothing, green or red light. Compliant or not
compliant situation. I'm sick of people sweeping what they interpret as
unimporant under the carpet only to learn down the track that those
points where in the spec for a good reason. The spec by test approach
would detect those problems early.
> --
> Regards,
>
> Tim Haughton
>
> Agitek - Agile Technology Solutions
> http://agitek.co.uk
> http://blogitek.com/timhaughton
Cheers.
Nick Malik [Microsoft]
> Your situation divides authority from responsibility. The specifying team
> doesn't have the responsibility to prove the designs. The remote team
> doesn't have the authority to improve the specifications. (And when tests
> drive development, writing tests in isolation is just like drawing models
> in
> isolation, and ordering someone else to implement the design.)
Excellent point. This is exactly why I asked on this forum. Thank you.
>
> To close this loop, the local team should implement at least 3 client
> applications for this framework.
Problem is that we are creating a small framework from scratch. To
implement a valid "application as a test" approach, the framework would have
to already exist. This is why I was discussing unit tests as specification,
not validation.
Once the framework is done, many apps will be based on it and I'm sure that
defects will be found.
Can we close the loop in another way? By emphasizing that unit tests can be
changed by the implementing team?
>
> And send a TDD coach to the remote team, to get them over the learning
> curve
> and show them out easy pure TDD is.
That isn't going to happen. Let's look for alternatives.
>
> Then share the responsibilities on both sides. Each team writes tests and
> code.
That's the core of the advice I'm taking away. Any followup to my comments?
Robert C. Martin
<nick...@hotmail.nospam.com> wrote:
>Should the coding team (in India) create the tests or should we create the
>tests and pass them over as a form of validation?
Both. You should specify the requirements in terms of automated
acceptance tests (e.g. take a look at fitnesse.org). The developers
(in india) should write the unit tests, and make all the tests pass.
-----
Robert C. Martin (Uncle Bob) | email: uncl...@objectmentor.com
Object Mentor Inc. | blog: www.butunclebob.com
The Agile Transition Experts | web: www.objectmentor.com
800-338-6716
"The aim of science is not to open the door to infinite wisdom,
but to set a limit to infinite error."
-- Bertolt Brecht, Life of Galileo
Roger L. Cauvin
news:vf0dd1tp1evjjmm3b...@4ax.com...
>
> You should specify the requirements in terms of automated
> acceptance tests (e.g. take a look at fitnesse.org). The
> developers (in india) should write the unit tests, and make
> all the tests pass.
Just a bit of an off-topic quibble. While I cannot disagree with the notion
of acceptance tests as specifications, sometimes they are not, and cannot
be, exactly equivalent to requirements. For details, see:
http://cauvin.blogspot.com/2005/07/should-all-requirements-be-testable.html
--
Roger L. Cauvin
nospam...@cauvin.org (omit the "nospam_" part)
Cauvin, Inc.
Product Management / Market Research
http://www.cauvin-inc.com
paul campbell
wrote:
> Just a bit of an off-topic quibble. While I cannot disagree with the
> notion
> of acceptance tests as specifications, sometimes they are not, and cannot
> be, exactly equivalent to requirements. For details, see:
>
> http://cauvin.blogspot.com/2005/07/should-all-requirements-be-testable.html
>
The one example you cite has no direct equivalent in software. You could
at least have made the effort to list one concrete software example.
Paul C.
Roger L. Cauvin
news:opsusock...@main.main...
The example of a super-reliable car is indeed outside the software realm.
Nevertheless, it is not practical to test directly whether the car will last
seven years. Note that it may be possible to simulate the seven year period
by subjecting the car to some accelerated wear and tear.
I'm disappointed that you weren't able to provide a software example
yourself. But thanks for challenging me; thinking about a software example
is helping to clarify my thoughts on this matter.
Imagine you're developing software to run a "marathon" car's engine. The
marathon car is a specialty car intended to be driven in a race continuously
for one year. The software governs various aspects of the engine so that it
runs more efficiently and with less wear over time. A nonfunctional
requirement of the software is that it must operate continuously for one
year. Just as with the nonsoftware example, we might indirectly be able to
test the requirement by, among other verification measures, ensuring the
program has no memory leaks. But it is not practical to test the constraint
directly; we would not want to wait one year to determine whether the
software satsifies it.
Phlip
space, not the other way around.
Requirements are things customers want, testable or not, and specifications
are specifics that submit to test. The quest to turn all requirements into
specifications is engineering. Some fit and some don't.
If you can't test it, it's not _specific_, so it's not a specification. For
example:
Requirement - the customer wants a car to run for a year.
Specification - engineers plan the tolerances for all the parts, then test
the parts satisfy them.
Oops, did I say "plan"? Scrub the mission. We plan the foam won't fall off.
The test is launching. We now return you to your regularily scheduled
traffic report.
--
Phlip
Robert C. Martin
<ro...@deadspam.com> wrote:
>The example of a super-reliable car is indeed outside the software realm.
>Nevertheless, it is not practical to test directly whether the car will last
>seven years. Note that it may be possible to simulate the seven year period
>by subjecting the car to some accelerated wear and tear.
The 7 yr mtbf cannot be a requirement because it cannot be proven. A
requirement is something that must be demonstrably true before
deployment. Anything else is a warranty, but not a requirement.
Roger L. Cauvin
> On Mon, 01 Aug 2005 01:37:34 GMT, "Roger L. Cauvin"
> <ro...@deadspam.com> wrote:
>
>> The example of a super-reliable car is indeed outside
>> the software realm. Nevertheless, it is not practical to
>> test directly whether the car will last seven years.
>> Note that it may be possible to simulate the seven year
>> period by subjecting the car to some accelerated wear
>> and tear.
>
> The 7 yr mtbf cannot be a requirement because it cannot
> be proven.
Well, it can in principle, but only after 7 years.
> A requirement is something that must be demonstrably
> true before deployment. Anything else is a warranty,
> but not a requirement.
We have two useful concepts here:
1. Requirements for product success
2. Requirements for product acceptance
The first concept is of utmost importance to product planners and
strategists (and ultimately to the customer). It is difficult to determine
or verify before the product is deployed (and sometimes even afterwards),
but is without a doubt a sensible and important concept. Product developers
rely on the second concept to drive design and implementation, and to
realistically determine when to release the product. For consulting
projects, sales people may also use it in contract negotiation.
You seem to be rejecting the first concept, or at least you somehow seem to
be trying to mandate that the word "requirement" not be used to describe it.
The fact is, however, we have two well-established and different terms in
product and software development: "requirements" and "acceptance criteria".
"Acceptance criteria" unambiguously denotes the second concept. A lot more
confusion surrounds the term "requirements", but most people seem to at
least sense that it isn't exactly synonymous with "acceptance criteria".
Why conflate the two terms when there are two separate - and useful and
important - concepts they can denote?
paul campbell
wrote:
> Imagine you're developing software to run a "marathon" car's engine. The
> marathon car is a specialty car intended to be driven in a race
> continuously
> for one year. The software governs various aspects of the engine so
> that it
> runs more efficiently and with less wear over time. A nonfunctional
> requirement of the software is that it must operate continuously for one
> year. Just as with the nonsoftware example, we might indirectly be able
> to
> test the requirement by, among other verification measures, ensuring the
> program has no memory leaks. But it is not practical to test the
> constraint
> directly; we would not want to wait one year to determine whether the
> software satsifies it.
Telemetry and control system such as this generally run on some sort
of descrete "tick" which represents the granularity of time they are
concerned with.
Simulating a full years run would simply be a case of running the
system faster-then-real-time using a much smaller tick (say 1/1000
of a second second instead of 1/100 or whatever) or even with zero
iteration delay.
Or there is a proof by induction approach i.e. we put the program in all
possible states that we know of and measure the memory usage - if it
flattens out after 1 minute we can infer it will work "forever".
Im not at all convinced that you are touching on anything deep or
profound (or even particularly challenging) with this observation of
yours.
Paul C.
paul campbell
On Sun, 31 Jul 2005 15:10:49 GMT, Roger L. Cauvin <ro...@deadspam.com>
wrote:
> "Robert C. Martin" <uncl...@objectmentor.com> wrote in message
> news:vf0dd1tp1evjjmm3b...@4ax.com...
>>
>> You should specify the requirements in terms of automated
>> acceptance tests (e.g. take a look at fitnesse.org). The
>> developers (in india) should write the unit tests, and make
>> all the tests pass.
>
> Just a bit of an off-topic quibble. While I cannot disagree with the
> notion
> of acceptance tests as specifications, sometimes they are not, and cannot
> be, exactly equivalent to requirements. For details, see:
>
> http://cauvin.blogspot.com/2005/07/should-all-requirements-be-testable.html
>
Just to add another sub discussion to this ....
My own personal favourite counter-example for specfication by tests
is a programming language.
However this is an extreme example and most business IT systems are
more linear in nature and thus ameanable to specification by test.
Paul C.
Roger L. Cauvin
> On Mon, 01 Aug 2005 01:37:34 GMT, Roger L. Cauvin <ro...@deadspam.com>
> wrote:
>
> Simulating a full years run would simply be a case of running
> the system faster-then-real-time using a much smaller tick
> (say 1/1000 of a second second instead of 1/100 or whatever)
> or even with zero iteration delay.
Yes, that would be one way of simulating the full year run. In general, we
strive to devise acceptance tests that come as close as possible to proving
that the product will satisfy the requirements. In some cases, we can test
the requirement directly. However, sometimes we settle for simulation or
other indirect means.
> Im not at all convinced that you are touching on anything deep
> or profound (or even particularly challenging) with this
> observation of yours.
That makes two of us :-) It's just important to keep in mind when
formulating requirements that, while we strive for measurability, this
measurability is to ensure clarity rather than direct translatability into
system tests.
Laurent Bossavit
> My own personal favourite counter-example for specfication by tests
> is a programming language.
How is that a counter-example ?
Laurent
Nick Malik [Microsoft]
> is a programming language.
>
> However this is an extreme example and most business IT systems are
> more linear in nature and thus ameanable to specification by test.
>
> Paul C.
I agree with Laurent. This is not a counter-example. In fact, I think it
to be an ideal example. A segment of higher-level language code must be
compiled into a specific set of instructions... or even to state that a
specific set of instructions must exhibit a specifically testable behavior.
Daniel Parker
language by tests?
-- Daniel Parker
Robert C. Martin
<daniel...@hotmail.com> wrote:
>Do you know of any prior experience with specifying a programming
>language by tests?
I'm no expert in this area, but it seems to me that there have been
standard test suites and benchmarks for C and C++. It would be very
interesting to design a language that passed these test suites, but
was not valid C or C++.
Laurent Bossavit
> Do you know of any prior experience with specifying a programming
> language by tests?
I'm not aware of any attempt to do exactly that, but there are languages
for which a comprehensive test suite was built in parallel with the
language; XSLT comes to mind.
It's much easier to find cases where a comprehensive test suite was
built after the fact, and is useful to the original implementors, to
vendors of "compliant" alternatives, and even to end users. Ada, Lisp,
Fortran, Java - most prominent languages have test suites.
Many TDD practitioners report, and my experience tends to confirm, that
developing comprehensive test suites after the fact and results in tests
that are harder to write and maintain, and which don't read as well. I
wouldn't be surprised if someone finally took the hint and wrote a
language test-first - not necessarily TDD, but at least building the
test suite systematically in parallel with the implementation.
"Specifying by tests" sounds a bit more ambitious, and ambiguous. There
are different areas to specify, more or less independent of each other;
the language syntax, the semantics with respect to some abstract
machine, the binary code generated, features such as garbage
collection...
Laurent
Daniel Parker
> On 2 Aug 2005 13:08:36 -0700, "Daniel Parker"
> <daniel...@hotmail.com> wrote:
>
>>Do you know of any prior experience with specifying a programming
>>language by tests?
>
> I'm no expert in this area, but it seems to me that there have been
> standard test suites and benchmarks for C and C++.
These test suites, though, are motivated by, and serve to validate, a
specification. The same goes for the XSLT specification. I follow the XSLT
mailing lists, and occaisonally still sample the C++ sample newsgroup. My
observation is that when there are questions about the validity of a
language implementation, the discussion is always with reference to the
specification, not the test suites.
> It would be very
> interesting to design a language that passed these test suites, but
> was not valid C or C++.
>
I'm sure you could find out by posting a question on comp.std.c++ :-)
Regards,
Daniel Parker
Daniel Parker
news:MPG.1d5a20c7f...@news.noos.fr...
>
> It's much easier to find cases where a comprehensive test suite was
> built after the fact, and is useful to the original implementors, to
> vendors of "compliant" alternatives, and even to end users. Ada, Lisp,
> Fortran, Java - most prominent languages have test suites.
Right, to validate a specification, not to be the specification, in some
sense.
>
> Many TDD practitioners report, and my experience tends to confirm, that
> developing comprehensive test suites after the fact and results in tests
> that are harder to write and maintain, and which don't read as well. I
> wouldn't be surprised if someone finally took the hint and wrote a
> language test-first - not necessarily TDD, but at least building the
> test suite systematically in parallel with the implementation.
>
> "Specifying by tests" sounds a bit more ambitious, and ambiguous.
But you suggest otherwise in your response to Paul Campbell.
> There
> are different areas to specify, more or less independent of each other;
> the language syntax, the semantics with respect to some abstract
> machine, the binary code generated, features such as garbage
> collection...
>
A specification of a modern programming language is based on theory. By
it's very nature, a specification is going to be a more economical
expression of the language than the suite of tests required to validate an
implementation. I would venture to guess that for the languages you refer
to above, there are many more people intimately familiar with the
specifications than the test suites. The test suites are important, it's
good that they're there, it's good that somebody took the trouble to write
them, but they're not normative, and most of us will never look at them.
They're more important to compiler implementors than to language designers.
-- Daniel Parker
Laurent Bossavit
> > "Specifying by tests" sounds a bit more ambitious, and ambiguous.
>
> But you suggest otherwise in your response to Paul Campbell.
I asked Paul a clarifying question; I don't know what you think I
"suggested".
Laurent
Dmitry A. Kazakov
> "Specifying by tests" sounds a bit more ambitious, and ambiguous. There
> are different areas to specify, more or less independent of each other;
> the language syntax, the semantics with respect to some abstract
> machine, the binary code generated, features such as garbage
> collection...
It does not sound any realistic. Apart from requirements like: the tasking
construct A does have race condition. [ Have a test for that? ]
But OK, just consider usual floating-point specification: for any two
operands X and Y, if the precise mathematical result of the operation X@Y
lies in the interval [L,U], where L and U are two adjacent machine numbers,
then the implementation of @ should return either L or U. Now, what would
be the test? To enumerate all floating-point numbers seems a minor problem
compared to, where to get that damned results of @?
Note that a validation suite is a very different thing than specification
by tests. If I know the behavior of @ in advance, if I know the
implementation of @, then I can relatively easily test *that*
implementation of *the* function with a good coverage.
--
Regards,
Dmitry A. Kazakov
http://www.dmitry-kazakov.de
Daniel Parker
language is a counter example to "specification by tests" because it
does not appear to be either realistic or desirable to specify a
non-trivial programming language entirely through tests.
Regards,
Daniel Parker
Laurent Bossavit
> I see, well then, the answer to your question is that a programming
> language is a counter example to "specification by tests" because it
> does not appear to be either realistic or desirable to specify a
> non-trivial programming language entirely through tests.
You've mustered one argument against "desirable". I've not seen an
argument against "realistic". I agree that the test suites you see with
languages that have one are not convincing examples of specifications of
the languages in question. That could be, largely, because they weren't
written with that in mind.
Laurent
Daniel Parker
> Daniel,
>
> > I see, well then, the answer to your question is that a programming
> > language is a counter example to "specification by tests" because it
> > does not appear to be either realistic or desirable to specify a
> > non-trivial programming language entirely through tests.
>
> You've mustered one argument against "desirable". I've not seen an
> argument against "realistic".
>
To suggest that a programming language could be specified through tests
alone is a fairly bold statement, and I would expect it to be
accompanied by some appeal to prior experience, or a sketch of how it
might work, but I don't see that, all I see is the statement.
-- Daniel
paul campbell
>
> "Specifying by tests" sounds a bit more ambitious, and ambiguous. There
> are different areas to specify, more or less independent of each other;
> the language syntax, the semantics with respect to some abstract
> machine, the binary code generated, features such as garbage
> collection...
I never meant to imply that it wasnt possible, but rather that a test
suite generally isnt the optimal thought vehicle for dealing with
such a conceptually dense and abstract subject. other examples would
be an AI that plays chess or a program to compose music. I would
tend to attack these sorts of problems in a much more abstract conceptual
manner than writting tests.
Its a far cry from capturing relatively linear concrete interactions
of business users with IT systems (which are very ameanable to spec
by test).
Lets say I wanted to write a program that could compose a harmonically
appropriate melody over the following chord sequence.
C F G C (repeat).
My conceptual specification would be:
"the duration of each note will be a quater-note"
"each sucessive candidate note will be randomly selected from the key
of C major."
"a candidate note will be rejected if that note
results in a group of three successive notes that do not correspond with
with a constituent note of the most recently played chord"
"a candidate note will be rejected if it is the second consecutive note
to change (from the last note) by a interval of more than a diatonic
forth"
"generation of candidate notes will continue until one is accepted, the
note will be added to the melody. This process will be repeated until
the end of the chord sequence is reached".
I'm really stumped as to how I would even start to write tests without
framing the strategy in at least in my mind first in those terms. But if
anyone fancies giving it a go ...
Paul C.
Paul C.
Robert C. Martin
<danielaparker@spam?nothanks.sympatico.ca> wrote:
>"Robert C. Martin" <uncl...@objectmentor.com> wrote in message
>news:alqve1hmn7chlamrl...@4ax.com...
>> On 2 Aug 2005 13:08:36 -0700, "Daniel Parker"
>> <daniel...@hotmail.com> wrote:
>>
>>>Do you know of any prior experience with specifying a programming
>>>language by tests?
>>
>> I'm no expert in this area, but it seems to me that there have been
>> standard test suites and benchmarks for C and C++.
>
>These test suites, though, are motivated by, and serve to validate, a
>specification.
Let's say that all major C++ compilers passed the X-Test, which was
written by the C++ standards org. If you were writing a C++ compiler
what would you use as your specification, the written spec or the
source code of the X-Test? If you followed the written spec and then
found that you failed to pass the X-test, what would you do? More
importantly, what would the standards committee do if they found that
the X-Test and the spec disagreed?
Robert C. Martin
<daniel...@hotmail.com> wrote:
>a programming
>language is a counter example to "specification by tests" because it
>does not appear to be either realistic or desirable to specify a
>non-trivial programming language entirely through tests.
Isn't it Noether's theorem that says that every specification implies
the existence a test?
<giggle>
Anyway, how do you test that a compiler conforms to the spec? You
convert the spec into tests and apply those tests (whether manually or
automatically, but hopefully automatically). This means that there is
a mapping of tests to specs, and specs back to tests. So specs *are*
tests, and tests *are* specs.
If we wrote the specs in a formal enough language, an automatic engine
could interpret them and test a compiler.
Robert C. Martin
<daniel...@hotmail.com> wrote:
>To suggest that a programming language could be specified through tests
>alone is a fairly bold statement, and I would expect it to be
>accompanied by some appeal to prior experience, or a sketch of how it
>might work, but I don't see that, all I see is the statement.
A compiler can be specified through tests. This is obvious on the
surface of it since a compiler is made up of executable lines of code,
and any executable line of code can be tested.
A compiler is not a language. However, since any spec is convertible
into a test and vice versa, it seems clear that a language can be
specified with tests.
Robert C. Martin
<paulc@.o.b.je.ct.vis.sion.co.uk> wrote:
>C F G C (repeat).
>
>My conceptual specification would be:
>
>"the duration of each note will be a quater-note"
>"each sucessive candidate note will be randomly selected from the key
>of C major."
>"a candidate note will be rejected if that note
>results in a group of three successive notes that do not correspond with
>with a constituent note of the most recently played chord"
>"a candidate note will be rejected if it is the second consecutive note
>to change (from the last note) by a interval of more than a diatonic
>forth"
>"generation of candidate notes will continue until one is accepted, the
>note will be added to the melody. This process will be repeated until
>the end of the chord sequence is reached".
>
>I'm really stumped as to how I would even start to write tests without
>framing the strategy in at least in my mind first in those terms. But if
>anyone fancies giving it a go ...
Test 1: Make sure the right number of notes is generated.
Make this pass.
Test 2: Make sure each note is a constituent of its corresponding
chord
Make this pass.
Test 3: Make sure that there are no consecutive intervals greater than
diatomic fourths.
Make this pass.
Test 4: Generate 100 melodies from the same chord sequence and ensure
that each note position is suitably random. (i.e. build a histogram
for each position, or just do a simple min-max test).
Make this pass.
paul campbell
Thanks, Ive probbaly made that example too easy in hindsight :-).
Paul C.
Dmitry A. Kazakov
> On 3 Aug 2005 06:30:44 -0700, "Daniel Parker"
> <daniel...@hotmail.com> wrote:
>
>>a programming
>>language is a counter example to "specification by tests" because it
>>does not appear to be either realistic or desirable to specify a
>>non-trivial programming language entirely through tests.
>
> Isn't it Noether's theorem that says that every specification implies
> the existence a test?
>
> <giggle>
>
> Anyway, how do you test that a compiler conforms to the spec? You
> convert the spec into tests and apply those tests (whether manually or
> automatically, but hopefully automatically). This means that there is
> a mapping of tests to specs, and specs back to tests. So specs *are*
> tests, and tests *are* specs.
Yes, a mapping exists, but this tells little about the properties of the
mapping. Consider a language like Algol 68 which defines types real, long
real, long long real, ad infinitum. The language also defines a literal Pi
for each such type. [ Clearly Pi is implemented as a function. ] Now, how
many tests are needed to specify the above? Firstly it is uncountable, and
secondly, no machine ever can run all of them.
This example shows the point: tests alone are quite weak in descriptive
power. Main weakness of testing is that you need a system sufficiently more
powerful than the object system, a system of a next class. In the case of a
language, the object system is *all possible programs*. Because all
languages are sort of finite-Turing complete, to specify one, you have to
be ve-e-e-ry patient or better, to get something beyond Turing... Language
designers [usually] take their heads! (:-))
BTW, tests as specifications could be extremely useful, but only as a
supplement to some a priori knowledge. Example: in mathematics
interpolation is nothing, but definition of a function from a *specified*
class using tests = interpolation points.
> If we wrote the specs in a formal enough language, an automatic engine
> could interpret them and test a compiler.
See above.
hans...@hotmail.com
> Anyway, how do you test that a compiler conforms to the spec? You
> convert the spec into tests and apply those tests (whether manually or
> automatically, but hopefully automatically). This means that there is
> a mapping of tests to specs, and specs back to tests. So specs *are*
> tests, and tests *are* specs.
It seems like your logic is flawed - correct me if I'm wrong. When
you state that 'You convert the spec into tests ...' you have
already made the assumption that this is possible. Half of the
conclusion you draw later 'is' in fact the assumption you made.
I don't like to be nitpicky, but since the relationship between tests
and specifications (and requirements) is of significance in IT
development, it is important to avoid misunderstandings.
Regards,
Hans Ewetz
Daniel Parker
Robert C. Martin wrote:
> On 3 Aug 2005 12:58:47 -0700, "Daniel Parker"
> <daniel...@hotmail.com> wrote:
>
> >To suggest that a programming language could be specified through tests
> >alone is a fairly bold statement, and I would expect it to be
> >accompanied by some appeal to prior experience, or a sketch of how it
> >might work, but I don't see that, all I see is the statement.
>
> A compiler can be specified through tests. This is obvious on the
> surface of it since a compiler is made up of executable lines of code,
> and any executable line of code can be tested.
>
> A compiler is not a language. However, since any spec is convertible
> into a test and vice versa, it seems clear that a language can be
> specified with tests.
>
(i.) Do you see a specification as being complete, in the sense that a
development team should be able to build a computer program from the
specification, without supplementary written or verbal communication?
(ii.) Do you regard a specification as a set of criteria that are
necessary and sufficient for verifying that a computer program is
correct?
(iii.) Is it important that a specification be meaningful to
developers, suggesting algorithms and approaches, or is it only
important that it perform a validating role?
Regards,
Daniel Parker
Nick Malik [Microsoft]
news:1123163866.5...@z14g2000cwz.googlegroups.com...
>
> Robert C. Martin wrote:
>> On 3 Aug 2005 12:58:47 -0700, "Daniel Parker"
>> <daniel...@hotmail.com> wrote:
>>
>> >To suggest that a programming language could be specified through tests
>> >alone is a fairly bold statement, and I would expect it to be
>> >accompanied by some appeal to prior experience, or a sketch of how it
>> >might work, but I don't see that, all I see is the statement.
>>
>> A compiler can be specified through tests. This is obvious on the
>> surface of it since a compiler is made up of executable lines of code,
>> and any executable line of code can be tested.
>>
>> A compiler is not a language. However, since any spec is convertible
>> into a test and vice versa, it seems clear that a language can be
>> specified with tests.
>>
> What do you mean by a specification?
>
> (i.) Do you see a specification as being complete, in the sense that a
> development team should be able to build a computer program from the
> specification, without supplementary written or verbal communication?
That is the funniest thing I've read this week. I think you intended it to
be biting but I find the notion humorous that ANY specification, whether by
tests or by documents, in lieu of ongoing learning and analysis, could
possible describe a complex system (whether a compiler or a business
application or a commercial product, etc) with anywhere near enough detail,
in a timely manner, to allow a development team to hide in the closet while
coding. The notion is completely absurd on its face. It's Clown paint. I
cannot imagine anyone who would seriously suggest that a specification is
something that allows a program to be built in a vacuum.
>
> (ii.) Do you regard a specification as a set of criteria that are
> necessary and sufficient for verifying that a computer program is
> correct?
I'm not sure that tests are an EFFICIENT way to do this. I suppose it is
possible, but I can think of better ways to spend time.
>
> (iii.) Is it important that a specification be meaningful to
> developers, suggesting algorithms and approaches, or is it only
> important that it perform a validating role?
The biggest fallacy that I run across in the notion of "large documents =
good software" is the idea that the details of a solution should be
described in a human language. I like documents for what they can do. You
can make a contract. You can describe a list of functions that your app
must support. You can describe costs and risks and effort plans. However,
the notion that requirements need to go through this process of creating
multiple specifications, one a parent to the next, until finally code pops
out... that is truly bizarre. One document for requirements. One for
design. Then code. If your designers are the only people smart enough to
understand how to build the code, why aren't they building the models that
produce the code? Writing a document is not a very effective way to teach,
and it is a sucky way to learn.
With respect and regards,
hans...@hotmail.com
> The biggest fallacy that I run across in the notion of "large documents =
> good software" is the idea that the details of a solution should be
> described in a human language. I like documents for what they can do. You
> can make a contract. You can describe a list of functions that your app
> must support. You can describe costs and risks and effort plans. However,
> the notion that requirements need to go through this process of creating
> multiple specifications, one a parent to the next, until finally code pops
> out... that is truly bizarre. One document for requirements. One for
> design. Then code. If your designers are the only people smart enough to
> understand how to build the code, why aren't they building the models that
> produce the code? Writing a document is not a very effective way to teach,
> and it is a sucky way to learn.
Are you suggesting that there is no need for multiple levels of
refinements between the initial requirements/domain description and the
code level?
By the way, actually writing a document about some subject is one of
the most efficient ways of learning about the subject.
Regards,
Hans Ewetz
Dmitry A. Kazakov
> "Daniel Parker" <daniel...@hotmail.com> wrote in message
> news:1123163866.5...@z14g2000cwz.googlegroups.com...
>>
>> (i.) Do you see a specification as being complete, in the sense that a
>> development team should be able to build a computer program from the
>> specification, without supplementary written or verbal communication?
[...]
> I cannot imagine anyone who would seriously suggest that a specification is
> something that allows a program to be built in a vacuum.
There are well- and ill-defined problems. It is quite possible to have a
problem exhaustively specified in one sentence.
> The biggest fallacy that I run across in the notion of "large documents =
> good software"
MSDN, is it what you mean? (sorry, couldn't resist (:-))
> ... is the idea that the details of a solution should be
> described in a human language. I like documents for what they can do. You
> can make a contract. You can describe a list of functions that your app
> must support. You can describe costs and risks and effort plans. However,
> the notion that requirements need to go through this process of creating
> multiple specifications, one a parent to the next, until finally code pops
> out... that is truly bizarre. One document for requirements. One for
> design. Then code. If your designers are the only people smart enough to
> understand how to build the code, why aren't they building the models that
> produce the code?
Yep, why should we study biology, genetics, organic chemistry, invest
billions in medicine, keep on failing with AI, if almost any uneducated
heterosexual pair can produce a child just in 9 months?
To be able /= to know how.
> Writing a document is not a very effective way to teach,
> and it is a sucky way to learn.
Oh, this is definitely wrong. Writing documents is how science works.
Document is a way to communicate ideas. It is a different, complementary to
the code way to do it. Programmers, who are unable convey their goals and
decisions in verbal form, are usually very bad coders. I saw many like
that. They keep on rewriting their program each time they face a minor
problem. Old versions are kept in the source commented out, just in case.
Of course they are unable to explain why new version should be any better
than the previous one. And guess what, it won't be better. This is typical
"reflex-driven developing". The best way I know to educate them, is to
forbid any further coding, until s/he would lay down a piece of paper
describing what and why s/he's going to do next.
Daniel Parker
Nick Malik [Microsoft] wrote:
> "Daniel Parker" <daniel...@hotmail.com> wrote in message
> news:1123163866.5...@z14g2000cwz.googlegroups.com...
> >
> >
> > development team should be able to build a computer program from the
> > specification, without supplementary written or verbal communication?
>
> That is the funniest thing I've read this week. I think you intended it to
> be biting
My intention is somewhat narrower. It's a question. Perhaps when
Robert Martin uses the term "tests as specification" he means something
very different from what most people regard as a specification. Would
he expect, for example, someone to be able to collect the test cases
for FitNesse, and without knowing anything else about FitNesse, be able
to reproduce something like FitNess from the test cases alone. Would
he expect this to be true more generally for other computer programs,
like compilers.
> but I find the notion humorous that ANY specification, whether by
> tests or by documents, in lieu of ongoing learning and analysis, could
> possible describe a complex system (whether a compiler
But presumably for a compiler the ongoing learning and analysis would
get reflected in the specification. Interpretations of the
specification, of course, would benefit from community discussion.
> or a business
> application or a commercial product, etc) with anywhere near enough detail,
> in a timely manner, to allow a development team to hide in the closet while
> coding. The notion is completely absurd on its face. It's Clown paint. I
> cannot imagine anyone who would seriously suggest that a specification is
> something that allows a program to be built in a vacuum.
>
I'm not suggesting that programs need be built this way, but rather,
could they be built this way, in principle, according to the "tests as
specification" people.
Regards,
Daniel Parker
Phlip
> Are you suggesting that there is no need for multiple levels of
> refinements between the initial requirements/domain description and the
> code level?
There is. That's why we sort features by business priority, and specify the
details of the important features before implementing them.
> By the way, actually writing a document about some subject is one of
> the most efficient ways of learning about the subject.
I don't want to burden my clients with the obligation to predict the future,
and guess which minimal subset of all these requirement ideas are the ones
that will lead directly to profit.
"How do I know? I'm not a psychic!" - Dr Cottle on Battlestar Galactica,
when asked whether Cmdr Adama would live.
--
Phlip
http://www.greencheese.org/ZeekLand <-- NOT a blog!!!
hans...@hotmail.com
> hansewetz wrote:
> > By the way, actually writing a document about some subject is one of
> > the most efficient ways of learning about the subject.
>
> I don't want to burden my clients with the obligation to predict the future,
> and guess which minimal subset of all these requirement ideas are the ones
> that will lead directly to profit.
Agree ... and your copmment is pretty orthogonal (to use one of your
own expressions) to my comment :-)
Regards,
Hans Ewetz
paul campbell
> On 3 Aug 2005 06:30:44 -0700, "Daniel Parker"
> <daniel...@hotmail.com> wrote:
>
>> a programming
>> language is a counter example to "specification by tests" because it
>> does not appear to be either realistic or desirable to specify a
>> non-trivial programming language entirely through tests.
>
> Isn't it Noether's theorem that says that every specification implies
> the existence a test?
>
> <giggle>
>
> Anyway, how do you test that a compiler conforms to the spec? You
> convert the spec into tests and apply those tests (whether manually or
> automatically, but hopefully automatically). This means that there is
> a mapping of tests to specs, and specs back to tests. So specs *are*
> tests, and tests *are* specs.
There is a mapping of every programming language construct to assembler but
that doesnt mean that it is desirable to code in assembler.
During this mapping intent is lost. Intent is also lost in the conversion
of
specs to tests and the more abstract and conceptually dense the subject
matter the more likely this is to happen.
It one thing to say that given any spec you can turn it into tests, but its
quite another to say that you can use tests alone to gain and communicate
understanding of the problem domain at the conceptual level.
Having said that there is a huge swath of problems that are ameanable to
such an approach but IMO it haas its limits in certain specialist areas.
Paul C.
Lurker
news:mu43f1h31fvvh9fcm...@4ax.com...
> You convert the spec into tests and apply those tests (whether manually
> or automatically, but hopefully automatically). This means that there is
> a mapping of tests to specs, and specs back to tests. So specs *are*
> tests, and tests *are* specs.
Let's take your typical syntax diagram for a programming language.
Given that, it's rather easy to write tests which validate that some
implementation passes those tests (note that it doesn't prove that the
implementation actually matches the specs - only that it passes your
tests). But that aside, how would you go with TDD first to come up
with that syntax diagram?
Nick Malik [Microsoft]
"Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> wrote in message
news:5j9lczkllsy6.1tjxet6ys3m4t$.dlg@40tude.net...
> On Thu, 4 Aug 2005 07:25:50 -0700, Nick Malik [Microsoft] wrote:
>
>> "Daniel Parker" <daniel...@hotmail.com> wrote in message
>> news:1123163866.5...@z14g2000cwz.googlegroups.com...
>>>
>>> (i.) Do you see a specification as being complete, in the sense that a
>>> development team should be able to build a computer program from the
>>> specification, without supplementary written or verbal communication?
>
> [...]
>> I cannot imagine anyone who would seriously suggest that a specification
>> is
>> something that allows a program to be built in a vacuum.
>
> There are well- and ill-defined problems. It is quite possible to have a
> problem exhaustively specified in one sentence.
For a small subset of the specifications in a single system of moderate
complexity, this statement would be 100% true. For the rest, a range of
further explanation and ongoing experimentation is needed to determine not
only what the requirements 'could' be but also what they 'should' be.
If you customer is standing next to you, you can show him or her two ideas,
perhaps giving a proof of concept bit for one or both, and then provide the
rest of the information: cost and performance tradeoffs, maintenance issues,
technical dependencies, etc, and let them decide which way to go. That may
involve the customer "changing" the requirements to suit their discovery of
not only what is possible but what is good. This collaboration is one of
the strengths of the agile processes.
This is why any document that attempts to describe an entire system is
simply a work of fiction. Useful fiction. Perhaps even somewhat
instructive fiction, but fiction all the same.
>
>> The biggest fallacy that I run across in the notion of "large documents =
>> good software" is the idea that the details of a solution should be
>> described in a human language. I like documents for what they can do.
>> [..] However,
>> the notion that requirements need to go through this process of creating
>> multiple specifications, one a parent to the next, until finally code
>> pops
>> out... that is truly bizarre. One document for requirements. One for
>> design. Then code. If your designers are the only people smart enough
>> to
>> understand how to build the code, why aren't they building the models
>> that
>> produce the code?
>
> Yep, why should we study biology, genetics, organic chemistry, invest
> billions in medicine, keep on failing with AI, if almost any uneducated
> heterosexual pair can produce a child just in 9 months?
>
> To be able /= to know how.
So you are stating that you have a developer who is able but does not know
how. Therefore someone who knows how should write it down in text, so that
the developer learns how. This is a fair statement.
If you note my text, I suggested that the person doing the writing should
create *models* that produce the code. I meant that literally. They should
be creating diagrams in a tool that, as much as possible, emits code. Then,
with fairly small amounts of text to explain context, the diagrams can be
used to illustrate and educate, while the emitted code can serve as an
initial framework for implementation. In this way, the ones who *know* how
to do it, actually do it. The rest learn by *collaboration*, not by
distribution of information through text.
>
>> Writing a document is not a very effective way to teach,
>> and it is a sucky way to learn.
>
> Oh, this is definitely wrong. Writing documents is how science works.
Hypothesis, Experimentation, and Analysis are how science works. Writing
documents may be how some students learn about some experiments. It is not
how scientists learn their craft. They learn from other scientists, in
collaboration. Where they cannot directly communicate, scientists will
REPEAT each other's experiments.
Therefore, if you want to compare the act of software specification to
science, then the designer has to CREATE the entire system, in software,
test it to make sure it works, describe it in text, and then the
implementors will read the text and repeat the 'experiment.' Most of use
are therefore more closely aligned with engineering than with science.
Computer science is our theoretical basis, just like physics is the
theoretical basis of mechanical engineering. In practice, we use the
information for different purposes.
>
> Document is a way to communicate ideas. It is a different, complementary
> to
> the code way to do it. Programmers, who are unable convey their goals and
> decisions in verbal form, are usually very bad coders.
Reread my statement. I did not say that no design document should be
written. I tend to agree with you that, in lieu of a design document, the
system is rubbish. The point is that we are saying the same things. I am
simply stating that it is disingenuous to sell the idea, to our superiors,
that you should take a requirements document, and create a functional
specification, and from that create a high-level-design document, and from
that create a detailed-level-design document, and from that create code...
and expect that the system will have improved substantially in the process.
It is OK to say that by producing a document, and then having smart people
inspect the document, the design may improve. I would state that by
producing code, and then having smart people inspect the code, the design is
likewise likely to improve. As long as the developer in question is not
emotionally attached to their code, and can change it as a result of
inspection, then inspecting the design, and then inspecting the code is the
most appropriate way to seek front-loaded quality improvement... not in
multiple layers of documentation.
> I saw many like
> that. They keep on rewriting their program each time they face a minor
> problem. Old versions are kept in the source commented out, just in case.
> Of course they are unable to explain why new version should be any better
> than the previous one. And guess what, it won't be better. This is typical
> "reflex-driven developing". The best way I know to educate them, is to
> forbid any further coding, until s/he would lay down a piece of paper
> describing what and why s/he's going to do next.
And then someone has to read the document, and offer feedback, and get them
to improve the document, and perhaps you would explain why a concept is bad,
in the document, by going to the whiteboard and illustrating an idea...
As long as there is one design document, we are in agreement. Requirements,
then design, then code. I find fault with the notion of "no design
document" as a response to "multiple design documents." You and I are
standing on common ground.
Roger L. Cauvin
news:R_6dndQ4xeR...@comcast.com...
>
> I am simply stating that it is disingenuous to sell the idea,
> to our superiors, that you should take a requirements
> document, and create a functional specification, and
> from that create a high-level-design document, and from that create a
> detailed-level-design document, and from
> that create code... and expect that the system will have
> improved substantially in the process.
I agree. Iterating on this process helps to mitigate scheduling,
architectural, and requirements risks. Agile processes have been in vogue
for some years now, even if most companies don't faithfully practice them.
However, many companies, even to the extent they use agile processes, seem
to iterate on the requirements. I.e., they take a requirements document as
a fixed given and iterate just on the design, coding, and testing. I
mention this tendency here:
http://cauvin.blogspot.com/2005/06/agile-product-management.html
--
Roger L. Cauvin
nospam...@cauvin.org (omit the "nospam_" part)
Cauvin, Inc.
Product Management / Market Research
http://www.cauvin-inc.com
Roger L. Cauvin
"However, many companies, even to the extent they use agile processes, DON'T
seem to iterate on the requirements."
"Roger L. Cauvin" <ro...@deadspam.com> wrote in message
news:CZKIe.99749$0f.3...@tornado.texas.rr.com...
Dmitry A. Kazakov
> "Dmitry A. Kazakov" <mai...@dmitry-kazakov.de> wrote in message
> news:5j9lczkllsy6.1tjxet6ys3m4t$.dlg@40tude.net...
>> Oh, this is definitely wrong. Writing documents is how science works.
>
> Hypothesis, Experimentation, and Analysis are how science works. Writing
> documents may be how some students learn about some experiments. It is not
> how scientists learn their craft.
You mean here individuals, not the scientific society. To make science as
an activity of many individuals from different countries and times working,
you need a framework within which they could communicate and verify their
results, i.e. scientific textbooks, magazines, conferences etc.
> Therefore, if you want to compare the act of software specification to
> science, then the designer has to CREATE the entire system, in software,
> test it to make sure it works, describe it in text, and then the
> implementors will read the text and repeat the 'experiment.' Most of use
> are therefore more closely aligned with engineering than with science.
SE is not yet engineering. Nevertheless, engineering is when you apply a
science. Science has a long tradition of requiring you to be able to
uphold your point of view. TDD lacks this, it turns directly to nature
(=computer & customer (:-)) and so misses the point.
> I am
> simply stating that it is disingenuous to sell the idea, to our superiors,
> that you should take a requirements document, and create a functional
> specification, and from that create a high-level-design document, and from
> that create a detailed-level-design document, and from that create code...
> and expect that the system will have improved substantially in the process.
But this exactly is how it *should* work, and it will work this way if and
when SE would some day become a true engineering. Presently we are in a
transitional state between craft and engineering. TDD is a way a craftsman
works. A cooper indeed needs no documentation, design etc. Test is an
excellent specification of a barrel...
>> I saw many like
>> that. They keep on rewriting their program each time they face a minor
>> problem. Old versions are kept in the source commented out, just in case.
>> Of course they are unable to explain why new version should be any better
>> than the previous one. And guess what, it won't be better. This is typical
>> "reflex-driven developing". The best way I know to educate them, is to
>> forbid any further coding, until s/he would lay down a piece of paper
>> describing what and why s/he's going to do next.
>
> And then someone has to read the document, and offer feedback, and get them
> to improve the document, and perhaps you would explain why a concept is bad,
> in the document, by going to the whiteboard and illustrating an idea...
Not always necessary, more important is to get people communicating their
ideas (even to themselves.) The document itself would likely be worthless.
Programmers are bad writers anyway.
> As long as there is one design document, we are in agreement.
Ideally, yes...
Phlip
Roger L. Cauvin wrote:
> I agree. Iterating on this process helps to mitigate scheduling,
> architectural, and requirements risks. Agile processes have been in vogue
> for some years now, even if most companies don't faithfully practice them.
> However, many companies, even to the extent they use agile processes,
don't
> seem
> to iterate on the requirements. I.e., they take a requirements document
> as
> a fixed given and iterate just on the design, coding, and testing. I
> mention this tendency here:
>
> http://cauvin.blogspot.com/2005/06/agile-product-management.html
Early Agile literature decried "Big Design Up Front", based on the
revelations that emergent design produced better results.
It turns out that BDUF might be less risky than "Big Requirements Up Front".
Studies like those cited in /Agile & Iterative Development/ by Craig Larman
show a direct correlation between the total volume of requirements
documented before coding and the odds that the speculated features never get
used. Such features are often cancelled, neglected, or reworked.
The thought experiment for this effect is simple: Any documentating effort
requires making decisions, and decisions made in isolation from feedback
increase risk.
Roger L. Cauvin's blog needs to mention that "just in time" requirement
analysis works best in order of business priority. The most important
features are, by nature, easiest to specify, and easiest to review we got
them right. Allowing them to lead analysis of lessor features reduces the
odds of rework; of revoking a feature and implementing another one.
hans...@hotmail.com
> Early Agile literature decried "Big Design Up Front", based on the
> revelations that emergent design produced better results.
The word 'agile' must be one of the most misused words in IT
discussions. XP proponents have decided that the meaning of the term
'agile' is directly related to one specific type of development
technique. It may come as a surprise, but there are multiple ways of
developing software in an 'agile' fashion. The XP way works
sometimes. Sometimes spending weeks or possibly months in understanding
a problem without producing any software whatsoever, allows for agile
development over the lifetime of the software. Unfortunately it is
difficult to know before hand if it is more 'agile' to spend time
modeling and doing analysis instead of going for an XP type approach.
This leads to the following question: what on earth is 'agile'
literature???????
Regards,
Hans Ewetz
Phlip
>
>> Early Agile literature decried "Big Design Up Front", based on the
>> revelations that emergent design produced better results.
>
> The word 'agile' must be one of the most misused words in IT
An Agile project can...
- accept feature requests in any order
- release any integration
- minimize the time between specifying a feature and
letting users profit from it.
> This leads to the following question: what on earth is 'agile'
> literature???????
Literature that jumped on the bandwagon? Literature with "Agile" on the
cover?
Literature that dodges and weaves easily?
Lurker
> The 7 yr mtbf cannot be a requirement because it cannot be proven. A
> requirement is something that must be demonstrably true before
> deployment. Anything else is a warranty, but not a requirement.
Well, it can be a requirement and it can be proven provided
you are willing to wait out a few hundred examples to see what
the *mean* time is.
If the point is that it must be "demonstrably true before deployment"
then it seems to fly in the face of TDD.
Phlip
> Robert C. Martin wrote:
>> The 7 yr mtbf cannot be a requirement because it cannot be proven. A
>> requirement is something that must be demonstrably true before
>> deployment. Anything else is a warranty, but not a requirement.
>
> Well, it can be a requirement and it can be proven provided
> you are willing to wait out a few hundred examples to see what
> the *mean* time is.
>
> If the point is that it must be "demonstrably true before deployment"
> then it seems to fly in the face of TDD.
Do you know of a methodology that handles untestable requirements?
Phlip
> The 7 yr mtbf cannot be a requirement because it cannot be proven. A
> requirement is something that must be demonstrably true before
> deployment. Anything else is a warranty, but not a requirement.
A great way to force in the untestable requirements is a burn-down chart. If
you require X but can't think of a test for it (such as MTBF), you specify
"management tests", which are metrics based on the code's current quality.
If you can simulate a week's workload in an hour's test, you can gather its
metrics and put these into a burn-down chart.
The team will deliberately manage to the chart, forcing its line down (or
up) over time. This is "Test Driven Management", where you set a goal that
within half a quarter you will have half your goal. The trend of the
burn-down chart will indicate where you can have all your goal within a
quarter.
Robert C. Martin
wrote:
I don't follow. If a requirement does not have to be verified prior
to deployment, then it must not be a true requirement.
TDD is a discipline that unambiguously verifies all requirements prior
to deployment.
Robert C. Martin
<daniel...@hotmail.com> wrote:
>
>Robert C. Martin wrote:
>> On 3 Aug 2005 12:58:47 -0700, "Daniel Parker"
>> <daniel...@hotmail.com> wrote:
>>
>> >To suggest that a programming language could be specified through tests
>> >alone is a fairly bold statement, and I would expect it to be
>> >accompanied by some appeal to prior experience, or a sketch of how it
>> >might work, but I don't see that, all I see is the statement.
>>
>> A compiler can be specified through tests. This is obvious on the
>> surface of it since a compiler is made up of executable lines of code,
>> and any executable line of code can be tested.
>>
>> A compiler is not a language. However, since any spec is convertible
>> into a test and vice versa, it seems clear that a language can be
>> specified with tests.
>>
>What do you mean by a specification?
>
>(i.) Do you see a specification as being complete, in the sense that a
>development team should be able to build a computer program from the
>specification, without supplementary written or verbal communication?
Ideally, yes.
>
>(ii.) Do you regard a specification as a set of criteria that are
>necessary and sufficient for verifying that a computer program is
>correct?
Yes.
>(iii.) Is it important that a specification be meaningful to
>developers, suggesting algorithms and approaches, or is it only
>important that it perform a validating role?
I prefer that a specification not suggest algorithms and approaches.
A spec should be a declarative statement of what is needed. It should
be written in a formalism that both business people and developers can
understand. It should be easily transformable into automated tests.
Robert C. Martin
<nick...@hotmail.nospam.com> wrote:
>"Daniel Parker" <daniel...@hotmail.com> wrote in message
>news:1123163866.5...@z14g2000cwz.googlegroups.com...
>>
>> (i.) Do you see a specification as being complete, in the sense that a
>> development team should be able to build a computer program from the
>> specification, without supplementary written or verbal communication?
>
>That is the funniest thing I've read this week. I think you intended it to
>be biting but I find the notion humorous that ANY specification, whether by
>tests or by documents, in lieu of ongoing learning and analysis, could
>possible describe a complex system (whether a compiler or a business
>application or a commercial product, etc) with anywhere near enough detail,
>in a timely manner, to allow a development team to hide in the closet while
>coding. The notion is completely absurd on its face. It's Clown paint. I
>cannot imagine anyone who would seriously suggest that a specification is
>something that allows a program to be built in a vacuum.
Nick, I think you are suffering from waterfall thinking. Imagine that
the spec is being written concurrently with the application, and leads
the application by a day or two. Moreover, imagine that there is a
high degree of feedback between the developers and the spec writers.
The result is still a spec that drives the developers (and the
feedback). Moreover, if the spec is written as automated tests, the
spec becomes an accurate description of the system.
Completeness is an issue of course. It requires a significant amount
of discipline to encode all the details of all the features as
customer acceptance tests. So it is likely that completeness will be
measured by degree, not as a boolean variable.
On the other and, if the developers are following the discipline of
TDD, and refusing to write any production code unless there is a
failing unit test first, then degree of completeness in the unit tests
will be very high indeed.
>> (ii.) Do you regard a specification as a set of criteria that are
>> necessary and sufficient for verifying that a computer program is
>> correct?
>
>I'm not sure that tests are an EFFICIENT way to do this. I suppose it is
>possible, but I can think of better ways to spend time.
A formal specification should be transformable into a test. If the
specification is suitably formal the transformation to a test could be
automatic. Indeed, one might not be able to tell the difference.
Each spec *would be* a test.
>> (iii.) Is it important that a specification be meaningful to
>> developers, suggesting algorithms and approaches, or is it only
>> important that it perform a validating role?
>The biggest fallacy that I run across in the notion of "large documents =
>good software" is the idea that the details of a solution should be
>described in a human language.
You mean a human conversational lan gauge, like English. However, all
languages that we use in Software are "human" languages. C++ is a
human language. It was written by humans for humans. This is not a
trivial point. There are human languages that can be designed to
facilitate the writing of formal specs as tests.
>However,
>the notion that requirements need to go through this process of creating
>multiple specifications, one a parent to the next, until finally code pops
>out... that is truly bizarre.
I agree that the waterfall workflow is bizarre. Doing *all* the
requirements, then *all* the analysis (whatever *that* is), then *all*
the design, and then *all* the code is simply not a good way to work.
It's inaccurate, inefficient, and (in the end) ineffective.
However, I think it is plainly obvious that specifications go through
stages of refinement until they are eventually transformed into code.
In my humble view they go from informal requirements to formal
requirements (tests) to designs, and code. The transformation is not
linear, it is dominated by feedback, and is therefore concurrent. All
stages mature together, rather than any stage completing before the
next begins.
Robert C. Martin
>"How do I know? I'm not a psychic!" - Dr Cottle on Battlestar Galactica,
>when asked whether Cmdr Adama would live.
That was a good episode. I'm surprised to find that I am really
enjoying that show.
Robert C. Martin
>Phlip wrote:
>
>> Early Agile literature decried "Big Design Up Front", based on the
>> revelations that emergent design produced better results.
>
>The word 'agile' must be one of the most misused words in IT
>discussions.
No more misused than "Object, Structured, Component, Modular, or any
of the other common prefixes to the trio
{Analysis|Design|Programming}.
>XP proponents have decided that the meaning of the term
>'agile' is directly related to one specific type of development
>technique.
The definition of the term "agile" as is pertains to software
development methods is quite clearly stated in the agile manifesto.
(www.agilemanifesto.org) Those of us who have adopted the practices
of XP find that the correlation between the agile manifesto, and the
values of XP, are very high.
>It may come as a surprise, but there are multiple ways of
>developing software in an 'agile' fashion. The XP way works
>sometimes. Sometimes spending weeks or possibly months in understanding
>a problem without producing any software whatsoever, allows for agile
>development over the lifetime of the software.
One of the primary values of the Agile Manifesto, which is the
definitive statement of the meaning of the word "agile" as used in the
context of "agile software development methods" is that the method
must be based on short cycles. So, given this definition of "agile",
a process that spends weeks or months understanding a problem without
delivering anything is NOT "agile".
>Unfortunately it is
>difficult to know before hand if it is more 'agile' to spend time
>modeling and doing analysis instead of going for an XP type approach.
Since the definition of the term "agile" specifies "short cycles",
this statement is untrue. To qualify as "Agile" you must break your
project into very small deliveries and perform your analysis and
design in those increments.
>
>This leads to the following question: what on earth is 'agile'
>literature???????
Much of it can be found at www.agilealliance.org
Robert C. Martin
<ro...@deadspam.com> wrote:
>
>"However, many companies, even to the extent they use agile processes, DON'T
>seem to iterate on the requirements."
I disagree. At least among those that I have encountered,
requirements are very much part of the iterative cycle. Indeed,
iterating the requirements it the most critical part of an agile
method in many cases.
It was Fred Brooks who said (and I paraphrase): "The document-driven,
specify-then-build approach, lies at the heart of so many software
problems." He was speaking specifically about requirements.
Robert C. Martin
<daniel...@hotmail.com> wrote:
>My intention is somewhat narrower. It's a question. Perhaps when
>Robert Martin uses the term "tests as specification" he means something
>very different from what most people regard as a specification. Would
>he expect, for example, someone to be able to collect the test cases
>for FitNesse, and without knowing anything else about FitNesse, be able
>to reproduce something like FitNess from the test cases alone. Would
>he expect this to be true more generally for other computer programs,
>like compilers.
Yes, I expect that would be the case. If you could take all 150 test
cases for FitNesse and make them pass with a brand new program, you
should end up with a program that is very much like FitNesse in it's
behavior.
Robert C. Martin
<mai...@dmitry-kazakov.de> wrote:
>Yes, a mapping exists, but this tells little about the properties of the
>mapping. Consider a language like Algol 68 which defines types real, long
>real, long long real, ad infinitum. The language also defines a literal Pi
>for each such type. [ Clearly Pi is implemented as a function. ] Now, how
>many tests are needed to specify the above? Firstly it is uncountable, and
>secondly, no machine ever can run all of them.
Which means that no compiler is correct because there exist legal
programs that every compiler will fail to compile. I find it
uninteresting to try to test the correctness of an untestable
construct. I also find that an untestable construct cannot be a
requirement; and is little more than a suggestion.
Robert C. Martin
<mai...@dmitry-kazakov.de> wrote:
>This example shows the point: tests alone are quite weak in descriptive
>power.
Yes, that's true. Tests are constrained to describe systems that can
be proven correct. They cannot describe systems that are unprovable
or inconsistent (like the Algol spec). I consider this to be an
advantage.
Robert C. Martin
>Robert C. Martin wrote:
>
>> Anyway, how do you test that a compiler conforms to the spec? You
>> convert the spec into tests and apply those tests (whether manually or
>> automatically, but hopefully automatically). This means that there is
>> a mapping of tests to specs, and specs back to tests. So specs *are*
>> tests, and tests *are* specs.
>
>It seems like your logic is flawed - correct me if I'm wrong. When
>you state that 'You convert the spec into tests ...' you have
>already made the assumption that this is possible. Half of the
>conclusion you draw later 'is' in fact the assumption you made.
>
>I don't like to be nitpicky, but since the relationship between tests
>and specifications (and requirements) is of significance in IT
>development, it is important to avoid misunderstandings.
Agreed. So I define a "requirement" as a attribute of a system that
can be proven to exist. If the attribute cannot be proven to exist,
then it is not required. It seems to me that this is axiomatic.
Therefore, any requirement can be proven. The proof of a requirement
is a test. Therefore any list of requirements (e.g. a spec) can be
mapped to an equivalent set of tests.
There are, of course, statements that appear to be requirements, but
are not. For example Dmitry's example algol in which the word 'long'
can prefix the word 'real' and unlimited number of times. While
'cute', this leaves the language is an ambiguous state since there are
perfectly legal programs that no compiler will ever be able to
compile. As such, this cannot be a requirement, and so the spec is
inconsistent.
Robert C. Martin
<paulc@.o.b.je.ct.vis.sion.co.uk> wrote:
>There is a mapping of every programming language construct to assembler but
>that doesnt mean that it is desirable to code in assembler.
>During this mapping intent is lost. Intent is also lost in the conversion
>of
>specs to tests and the more abstract and conceptually dense the subject
>matter the more likely this is to happen.
I completely disagree. Specifications are highly detailed declarative
statements that do not include intent. Intent is always commentary.
The declarative statements of the spec are transformable into tests
without loss of intent, because there is no intent in those
statements.)
Robert C. Martin
Dmitry A. Kazakov
> On Thu, 4 Aug 2005 10:23:20 +0200, "Dmitry A. Kazakov"
> <mai...@dmitry-kazakov.de> wrote:
>
>>This example shows the point: tests alone are quite weak in descriptive
>>power.
>
> Yes, that's true. Tests are constrained to describe systems that can
> be proven correct.
Not quite. In fact you mean "will be built", you equalize
provable=testable.
> They cannot describe systems that are unprovable
> or inconsistent (like the Algol spec). I consider this to be an
> advantage.
An immediate analogy: that would be a kind of constructivism for software.
Though more restrictive and far weaker than in mathematics, because you
tossing away even induction. My example with Pi is provable, but not
testable (using brute force.) Though we can formally prove both that an
algorithm works for any given finite precision, and that the given program
implements the algorithm. BTW, the latter can be done by tests.
There is a large class of similar situations, where we have to jump over
infinity and non-testability to come to something finite and testable.
Starting with a program adding numbers of indefinite precision and ending
with concurrent real-time systems.
The idea that something non-testable cannot be a requirement is attractive,
but unrealistic. Consider year 2000 bug. It is non-testable to warranty
your system be working in the next 5 years of *real* time. Now say the
customer, that you'd like to withdraw this requirement. Just test this!
(:-))
Phlip
> Phlip wrote:
>
>>"How do I know? I'm not a psychic!" - Dr Cottle on Battlestar Galactica,
>>when asked whether Cmdr Adama would live.
>
> That was a good episode. I'm surprised to find that I am really
> enjoying that show.
Besides relentlessly turning Space Opera cliche's on their heads, it was
also a nod to the running Star Trek joke that started with Bones saying "I'm
a doctor, not a coal miner"...
Lurker
> >Let's take your typical syntax diagram for a programming language.
> >Given that, it's rather easy to write tests which validate that some
> >implementation passes those tests (note that it doesn't prove that the
> >implementation actually matches the specs - only that it passes your
> >tests). But that aside, how would you go with TDD first to come up
> >with that syntax diagram?
>
> One syntax element at a time.
That's a glib answer that doesn't really answer anything at all.
Let's take a simple, well-defined language like Pascal say.
Its language diagram is trivial, but how would you arive at
it with tests first?
hans...@hotmail.com
> On 5 Aug 2005 09:03:05 -0700, hans...@hotmail.com wrote:
> >It may come as a surprise, but there are multiple ways of
> >developing software in an 'agile' fashion. The XP way works
> >sometimes. Sometimes spending weeks or possibly months in understanding
> >a problem without producing any software whatsoever, allows for agile
> >development over the lifetime of the software.
>
> One of the primary values of the Agile Manifesto, which is the
> definitive statement of the meaning of the word "agile" as used in the
> context of "agile software development methods" is that the method
> must be based on short cycles. So, given this definition of "agile",
> a process that spends weeks or months understanding a problem without
> delivering anything is NOT "agile".
My point was that there are ways of developing software in an 'agile'
way that do not introduce short cycles. You are confusing agility in
software development with the method for how you achieve it.
> >Unfortunately it is
> >difficult to know before hand if it is more 'agile' to spend time
> >modeling and doing analysis instead of going for an XP type approach.
>
> Since the definition of the term "agile" specifies "short cycles",
> this statement is untrue. To qualify as "Agile" you must break your
> project into very small deliveries and perform your analysis and
> design in those increments.
Let's clarify - to qualify as 'agile' according to the 'agile
manifesto' you must ...'
It seems as if you mist the point I was trying to make: XP type of
development is but one way of ensuring agility in software development.
Regards,
Hans Ewetz
Lurker
> I completely disagree. Specifications are highly detailed declarative
> statements that do not include intent. Intent is always commentary.
Let me see if I got it right. Acording to you, specifications
do not include intent? So what are they for then? I mean,
the whole point of the specs is to communicate the intent,
isn't it?
ggr...@bigfoot.com
This is for Lurker, and also for Daniel Parker and Dmitry Kazakov :
We have SPEC, which is a specification of something for which an
implementation IMPL is required. We have TEST, which is a set of
tests that attempt to show that IMPL conforms to SPEC.
The premise is that there is a function f such that :
1. f(SPEC) = TEST
Succinctly, f allows the derivation of a set of tests from a spec.
2. Now what we continually hear from the XP/TDD is :
The tests are the specification.
3. From 1, 2 implies (SPEC = TEST)
4. Lurker IMHO is asking the following question :
Given f-inverse(T) = S, SPEC = some programming language,
show us the elements of TEST such that we can determine whether
f-inverse(TEST) = SPEC.
Given that a programming language may be too complex for debate,
what example would someone suggest ??
Late last yr I proposed the game of Go, where I would write SPEC
(using mathematical techniques) for the game, and an XP bod would write
TEST as "customer" tests. The premise being that what was specified/
omitted/incorrect using either approach could be scrutinised. That one
soon went quiet.
Shall we be topical, and chose something like Depth First Search of
directed graphs ??
Something else ?? I (and no doubt others) will be open to offers.
Regards,
Steven Perryman
PS: I have more interest than others in SPEC and f, because it is
the basis of the specification-directed testing method. Similarly for
f-inverse (Design for Testability) , because it leads to the creation
of output-independent test infrastructure.
Roger L. Cauvin
> On Fri, 05 Aug 2005 14:51:56 GMT, "Roger L. Cauvin"
> <ro...@deadspam.com> wrote:
>
>>
>>"However, many companies, even to the extent they use
>> agile processes, DON'T seem to iterate on the requirements."
>
> I disagree. At least among those that I have encountered,
> requirements are very much part of the iterative cycle. Indeed,
> iterating the requirements it the most critical part of an agile
> method in many cases.
>
> It was Fred Brooks who said (and I paraphrase): "The
> document-driven, specify-then-build approach, lies at the
> heart of so many software problems." He was speaking
> specifically about requirements.
You're preaching to the choir, Robert. I'm all for iterating on
requirements. Indeed, my company encourages clients to incorporate our
product management services, including refining and refactoring
requirements, in an iterative product development process. I just don't see
most companies using such an approach (unfortunately).
--
Roger L. Cauvin
nospam...@cauvin.org (omit the "nospam_" part)
Cauvin, Inc.
Product Management / Market Research
http://www.cauvin-inc.com
Laurent Bossavit
> > One syntax element at a time.
>
> That's a glib answer that doesn't really answer anything at all.
>
> Let's take a simple, well-defined language like Pascal say.
> Its language diagram is trivial, but how would you arive at
> it with tests first?
You might not arrive anywhere near Pascal.
Let's get the ball rolling on the experiment. Below is a test you could
implement fairly easily in Fit:
Expression Value
3 3
The above test says that in TDD-Lang, the constant expression "3"
evalutates to the integer value 3. That's one syntax element. What do
you want as the next one ?
Laurent
Roger L. Cauvin
>
> I don't follow. If a requirement does not have to be
> verified prior to deployment, then it must not be a true
> requirement.
>
> TDD is a discipline that unambiguously verifies all
> requirements prior to deployment.
I agree with the thrust of your comments here about TDD. We simply have a
difference in what we mean by "requirement".
As I mentioned in a different message, I think you're conflating
"requirement" with "acceptance criterion". All acceptance criteria should
be directly testable before deployment. They should also reflect the
requirements. But requirements and acceptance criteria are not exactly the
same thing.
Requirements are testable, but not necessarily directly testable before
deployment. Sometimes we translate requirements into acceptance criteria
that are directly testable before deployment. To illustrate, let's look
again at my "marathon car" example:
>>> Imagine you're developing software to run a "marathon"
>>> car's engine. The marathon car is a specialty car intended
>>> to be driven in a race continuously for one year. The
>>> software governs various aspects of the engine so that it
>>> runs more efficiently and with less wear over time. A
>>> nonfunctional requirement of the software is that it must
>>> operate continuously for one year. Just as with the
>>> nonsoftware example, we might indirectly be able to test
>>> the requirement by, among other verification measures,
>>> ensuring the program has no memory leaks. But it is not
>>> practical to test the constraint directly; we would not want
>>> to wait one year to determine whether the software
>>> satsifies it.
The requirement that the software must operate continuously for one year is
not directly testable before deployment (unless we're willing to wait at
least a year). However, as Paul Campbell pointed out, we can devise
acceptance criteria that indirectly verify the requirement with some level
of confidence. His proposal was:
>> Simulating a full years run would simply be a case of running
>> the system faster-then-real-time using a much smaller tick
>> (say 1/1000 of a second second instead of 1/100 or whatever)
>> or even with zero iteration delay.
We can't just pretend that the requirement that the software run
continuously for a year doesn't exist just because it's not practically
possible to test directly before deployment. Instead, we should embrace the
requirement and devise acceptance tests that verify it with a high degree of
confidence.
Laurent Bossavit
> Late last yr I proposed the game of Go, where I would write SPEC
> (using mathematical techniques) for the game, and an XP bod would write
> TEST as "customer" tests. The premise being that what was specified/
> omitted/incorrect using either approach could be scrutinised. That one
> soon went quiet.
Let's start again... We ended up with the following candidate
requirements (and a critique of them from me, which never got a
response) :
) Provide a 19x19 board with the ability for two players to exchange
moves
2) Remove captured groups/stones automatically
3) Implement end-of-game rule (3 consecutive passes)
4) Implement the Ko rule
5) Allow default komi editable by player before a game (not during)
6) Support setting number of handicap stones
7) Support 13x13 and 9x9 board sizes (adjust max handicap accordingly)
8) Score the game at the end
9) Save game to file (SGF format?)
10) Replay game loaded from file (SGF format?)
11) Supply estimate of score while game is in progress
Laurent
hans...@hotmail.com
> "Robert C. Martin" <uncl...@objectmentor.com> wrote in message
> news:ckhcf19bde0rk63o3...@4ax.com...
> >
> > I don't follow. If a requirement does not have to be
> > verified prior to deployment, then it must not be a true
> > requirement.
> >
> > TDD is a discipline that unambiguously verifies all
> > requirements prior to deployment.
>
> I agree with the thrust of your comments here about TDD. We simply have a
> difference in what we mean by "requirement".
Of course there is confusion between "requirement" and "acceptance
criterion"!
After a few discussions on the group I have come to realize that TDD
sees requirements and tests as different views of the same concept.
>From their viewpoint there exist a concept, let's call it FOO, which
instances can be expressed either as a test, as a requirement or as
executable code. The purpose of expressing a FOO as a test is to be
able to check that the FOO instance is invariant under the
transformation that produced the executable code. The reason for
expressing a FOO instance as a requirement is to let someone express
how they want the software to behave.
The problem is that this type of view puts severe restrictions on what
business people can state as requirements. Business people usually
don't want to say anything about the software - they want to state
what should happen in the world when the software executes. To satisfy
TDD, the business people will have to reduce their wishes to a few
statements that describe the state of the world at specific points in
time. They have to do this since they know that the rules of TDD state
that it is not allowed to wish for something that cannot be transformed
into an executable test - and executable tests can only check a small
number of discrete states. If they want the software to sound a bell
every hour for the next 100 years, they cannot state this as a
requirement unless they are willing to wait 100 years.
I would propose to do away with the term 'requirement' when TDD is
used. It is redundant since the term 'test' carries an identical
meaning - that is, for TDD developers, not for all others. Or even
better, introduce the concept of a BAR - a BAR describes how the
business people wants the world to behave when the software is
executed. Now, the only thing we need to do is to transform a BAR into
a requirement ... sorry, a test. On the other hand, a BAR is what most
people refer to as a requirement ...
Regards,
Hans Ewetz
ggr...@bigfoot.com
Yes. I suggest requirements (REQ) = 1-8, 11.
So we have :
- You to write (in whatever form you see fit) , all the "customer"
tests
you need in order to verify that any submitted impl of the game of
Go conforms to REQ.
- Me to write a spec (in whatever form I see fit) that captures REQ
sufficiently well enough to enable an impl to be devised by a
developer.
1. I suggest deliverables grouped as follows :
1/2/3/4/8, 5/6/7, 11
This way we can see the artifact evolution thru the work activity.
2. We need the same requirements sources (online game rules etc) .
Info not available in those sources is not usable.
3. We need an independent source to hold the deliverables until we
have both finished. I do not want to be unfairly advantaged by seeing
your stuff thus potentially bettering what I might otherwise devise.
It will be interesting to see if I can define the whole thing so a
developer merely has to cut and paste my spec as Design By Contract
artifacts.
Regards,
Steven Perryman
Phlip
> Robert C. Martin wrote:
>
>> I completely disagree. Specifications are highly detailed declarative
>> statements that do not include intent. Intent is always commentary.
>
> Let me see if I got it right. Acording to you, specifications
> do not include intent? So what are they for then? I mean,
> the whole point of the specs is to communicate the intent,
> isn't it?
This is indeed the paradox. The ultimate goal of a software engineering
project is to communicate the client's intent into an application. Intent
cannot be tested, so the medium of communication must be testable
derivatives of the intent.
Trippy!
Phlip
> My point was that there are ways of developing software in an 'agile'
> way that do not introduce short cycles.
The three corners of this triangle are Waterfall, Code-and-Fix, and Agile.
Waterfall is the least constrained and least incremental. Code-and-Fix is
least constrained and randomly incremental. You release whenever, not
Friday.
Agile is the name we have claimed for the most constrained, regularily
incremental processes.
A given project might succeed from anywhere on the triangle. We simply
cannot advise anyone to try it like that; the success might not be as
reproducible as we have found regular iterations to provide.
> Let's clarify - to qualify as 'agile' according to the 'agile
> manifesto' you must ...'
Prefer interactions over paperwork, and release working software regularily.
This goes back to systems theory, how large things always grow in small
steps.
Phlip
> The problem is that this type of view puts severe restrictions on what
> business people can state as requirements.
That's a problem?
PHB to Dilbert: "I need a list of every acronym that nobody uses."
> Business people usually
> don't want to say anything about the software - they want to state
> what should happen in the world when the software executes. To satisfy
> TDD, the business people will have to reduce their wishes to a few
> statements that describe the state of the world at specific points in
> time.
The team does that. The programmers help the business side translate dreams
into tests, and the programmers teach the business side to create Acceptance
Tests. This leads to steering the project from where it is, not idle
speculation where it could be.
> I would propose to do away with the term 'requirement' when TDD is
> used.
That's why I always say "specification". As in "testers lead a TDD project
by collaborating with clients to convert requirements into testable
specifications."
Laurent Bossavit
> Yes. I suggest requirements (REQ) = 1-8, 11.
One reason I've called them "candidate requirements" is that I suspect
they should not all make it as written into the final specifications. In
particular, I'd want to see 8 broken down into smaller pieces. And it
seems to me that 1) belongs with 5) through 7) - there will be many
tests for which the board dimensions don't matter.
> 3. We need an independent source to hold the deliverables until we
> have both finished. I do not want to be unfairly advantaged by seeing
> your stuff thus potentially bettering what I might otherwise devise.
I'm not sure what's "unfair" about that. The point, as I see it, is to
explore different approaches, not to hold a contest.
Laurent
hans...@hotmail.com
> hans...@hotmail.com> wrote in message
>
> > The problem is that this type of view puts severe restrictions on what
> > business people can state as requirements.
>
> That's a problem?
Again, it all boils down to the definition of the term 'requirement'.
You have simply introduced a new definition for the term requirement in
TDD. This is OK and I don't really have a problem with that. However,
it should be clear that your definition of requirement is NOT the same
as the definition that many developers had before TDD was introduced to
the world.
> > Business people usually
> > don't want to say anything about the software - they want to state
> > what should happen in the world when the software executes. To satisfy
> > TDD, the business people will have to reduce their wishes to a few
> > statements that describe the state of the world at specific points in
> > time.
>
> The team does that. The programmers help the business side translate dreams
> into tests, and the programmers teach the business side to create Acceptance
> Tests. This leads to steering the project from where it is, not idle
> speculation where it could be.
OK, I'll rename my BAR to 'dreams'.
> > I would propose to do away with the term 'requirement' when TDD is
> > used.
>
> That's why I always say "specification". As in "testers lead a TDD project
> by collaborating with clients to convert requirements into testable
> specifications."
Hmmm ... I remember having a discussion about 'specification' before.
My understanding in the TDD world is that there is a 1-1 mapping
between specifications and tests. Let me loop back to the beginning of
my previous posting, but change the word 'requirement' to
'specification'.
Regards,
Hans Ewetz
ggr...@bigfoot.com
>> Yes. I suggest requirements (REQ) = 1-8, 11.
> One reason I've called them "candidate requirements" is that I suspect
> they should not all make it as written into the final specifications. In
> particular, I'd want to see 8 broken down into smaller pieces. And it
> seems to me that 1) belongs with 5) through 7) - there will be many
> tests for which the board dimensions don't matter.
What you call them, how you divide/refine them is moot.
Do we have agreement on 1-8, 11 being the start point for both of us ??
If so, we now need to agree the information sources for the subject
domain of the game of Go.
>> 3. We need an independent source to hold the deliverables until we
>> have both finished. I do not want to be unfairly advantaged by seeing
>> your stuff thus potentially bettering what I might otherwise devise.
> I'm not sure what's "unfair" about that. The point, as I see it, is to
> explore different approaches, not to hold a contest.
The point is to give an objective demonstration of work process.
If I am beneficially influenced by seeing your work, then it is not a
fair reflection on my work process.
Regards,
Steven Perryman
Laurent Bossavit
> I'm really stumped as to how I would even start to write tests without
> framing the strategy in at least in my mind first in those terms.
The point of writing the tests is to get the strategy out of your mind
and into some permanent medium. No one's suggesting you shouldn't try to
frame the strategy, mentally, before writing the tests.
> I never meant to imply that it wasnt possible, but rather that a test
> suite generally isnt the optimal thought vehicle for dealing with
> such a conceptually dense and abstract subject.
My answer to that is the same as in the case of a music program: "It
depends what kind of test suite." I'll readily admit that a long series
of assertEquals() might not be suitable, but that's only one way of
writing tests.
Laurent
hans...@hotmail.com
> That's why I always say "specification". As in "testers lead a TDD project
> by collaborating with clients to convert requirements into testable
> specifications."
I forgot to make one comment in my previous posting.
In the TDD world, are a set of requirements and the corresponding set
of specifications 'isomorphic'? The same question for
specifications and tests.
Regards,
Hans Ewetz
Phlip
> In the TDD world, are a set of requirements and the corresponding set
> of specifications 'isomorphic'?
Nyet!
> The same question for
> specifications and tests.
Uh, they are the same thing, so I think that qualifies for "isomorphic"...
hans...@hotmail.com
> On 4 Aug 2005 04:46:45 -0700, hans...@hotmail.com wrote:
>
> >Robert C. Martin wrote:
> >
> >> Anyway, how do you test that a compiler conforms to the spec? You
> >> convert the spec into tests and apply those tests (whether manually or
> >> automatically, but hopefully automatically). This means that there is
> >> a mapping of tests to specs, and specs back to tests. So specs *are*
> >> tests, and tests *are* specs.
> >
> >It seems like your logic is flawed - correct me if I'm wrong. When
> >you state that 'You convert the spec into tests ...' you have
> >already made the assumption that this is possible. Half of the
> >conclusion you draw later 'is' in fact the assumption you made.
> Agreed. So I define a "requirement" as a attribute of a system that
> can be proven to exist. If the attribute cannot be proven to exist,
> then it is not required. It seems to me that this is axiomatic.
Sounds solid
(attribute AND can-be-proven-to-exist => attribute-is-a-requirement)
> Therefore, any requirement can be proven. The proof of a requirement
> is a test. Therefore any list of requirements (e.g. a spec) can be
> mapped to an equivalent set of tests.
Agree. However, your definition does not help in terms of ensuring that
the software will work correctly. For example, a requirement on the
software for a Mars probe could be that the software must function
during the entire Mars flight. You can device a test that checks the
requirement ... you send a probe to Mars.
There is also the issue of what the terms 'proving' and 'test' means.
Depending on the exact meaning of the terms the conclusions will vary.
Regards,
Hans Ewetz
Laurent Bossavit
> Do we have agreement on 1-8, 11 being the start point for both of us ??
Sure...
> If so, we now need to agree the information sources for the subject
> domain of the game of Go.
As long as it's online...
> The point is to give an objective demonstration of work process.
> If I am beneficially influenced by seeing your work, then it is not a
> fair reflection on my work process.
My point of view would be that if I'm beneficially influenced by
anything I happen to come across, then so much the better. I'm in this
not so much to prove how smart I am but to get smarter. I'd be inclined
to post whatever I come up with, so that better heads than me can take
potshots at it.
Laurent
Ilja Preuß
>
>> My point was that there are ways of developing software in an 'agile'
>> way that do not introduce short cycles.
I think you will have to accept that "Agile Software Development" is a name,
one that refers to a way of doing software development that definitely
includes short cycles.
Take care, Ilja
Robert C. Martin
>Robert C. Martin wrote:
>> On 4 Aug 2005 04:46:45 -0700, hans...@hotmail.com wrote:
>> Agreed. So I define a "requirement" as a attribute of a system that
>> can be proven to exist. If the attribute cannot be proven to exist,
>> then it is not required. It seems to me that this is axiomatic.
>
>Sounds solid
>(attribute AND can-be-proven-to-exist => attribute-is-a-requirement)
>
>> Therefore, any requirement can be proven. The proof of a requirement
>> is a test. Therefore any list of requirements (e.g. a spec) can be
>> mapped to an equivalent set of tests.
>
>Agree. However, your definition does not help in terms of ensuring that
>the software will work correctly. For example, a requirement on the
>software for a Mars probe could be that the software must function
>during the entire Mars flight. You can device a test that checks the
>requirement ... you send a probe to Mars.
Agreed. And, as we've recently found, some probes have failed this
test; i.e. they did not meet their requirements. This just goes to
show that the tests for the system have to be as close to the
production environment as is practical.
Robert C. Martin
wrote:
No, the point of a spec is to communicate desired behavior. The
reason for the desire is not part of the formal spec, though it may be
included as a comment.
Robert C. Martin
wrote:
I'm not an expert of Pascal syntax. So let's try C.
assertTrue(isValidStatement("int i;"));
assertFalse(isValidStatement("x i;"));
Let's get this to pass. Then we can go on the next syntax element.
Roger L. Cauvin
news:1123509489....@f14g2000cwb.googlegroups.com...
>
> Of course there is confusion between "requirement" and
> "acceptance criterion"!
>
> After a few discussions on the group I have come to
> realize that TDD sees requirements and tests as different
> views of the same concept.
And in a way they are right.
> I would propose to do away with the term 'requirement'
> when TDD is used. It is redundant since the term 'test'
> carries an identical meaning - that is, for TDD developers,
> not for all others.
Hans, I think we can accomodate the established terminology ("requirement",
"acceptance criterion", "test") and still satisfy both TDD proponents and
opponents.
A requirement specifies a condition that must hold to solve or avoid a
problem a customer (or prospective customer) faces. As such, it is, or
directly implies, a test. The test may be impossible, for practical
reasons, to perform prior to release, but it is still possible to perform in
principle given sufficient time and resources.
An acceptance criterion specifies a condition that must hold before the
product is suitable for release. It also is, or directly implies, a test.
Indeed, in the realm of software you can express the test as code, which TDD
advocates. Traditionally, we call this test an "acceptance test". However,
an acceptance criterion is not the same as a requirement. While it is
possible to test a requirement in principle, it is not always possible to
test it directly prior to release. Hence we formulate acceptance criteria
(as tests, if we use TDD) that come as close as possible to guaranteeing
satisfaction of the requirements.
Roger L. Cauvin
>> For example, a requirement on the software for a
>> Mars probe could be that the software must function
>> during the entire Mars flight. You can device a test
>> that checks the requirement ... you send a probe to
>> Mars.
>
> Agreed. And, as we've recently found, some probes
> have failed this test; i.e. they did not meet their
> requirements. This just goes to show that the tests for
> the system have to be as close to the production
> environment as is practical.
Robert, this statement is perfectly reasonable and captures the distinctions
I tried to make upon entering this discussion thread. I cited a blog entry
that distinguished between "testability in principal" and "testability in
practice", and contended that requirements must be testable in principle,
but not necessarily directly and comprehensively testable in practice before
release.
Can we agree that requirements must, in principle, be testable to be
meaningful? Can we also agree that acceptance criteria must be directly
testable (prior to release) to be useful? Can we further agree that
acceptance criteria should come as close as possible to reflecting the
requirements, but that they are not necessarily the same?
Phlip
> I'm not an expert of Pascal syntax. So let's try C.
>
> assertTrue(isValidStatement("int i;"));
> assertFalse(isValidStatement("x i;"));
>
> Let's get this to pass. Then we can go on the next syntax element.
bool isValidStatement(char const * parseMe)
{
return 0 == strcmp(parseMe, "int i;");
Lurker
news:1123171799....@f14g2000cwb.googlegroups.com...
> My intention is somewhat narrower. It's a question. Perhaps when
> Robert Martin uses the term "tests as specification" he means something
> very different from what most people regard as a specification. Would
> he expect, for example, someone to be able to collect the test cases
> for FitNesse, and without knowing anything else about FitNesse, be able
> to reproduce something like FitNess from the test cases alone. Would
> he expect this to be true more generally for other computer programs,
> like compilers.
In a similar strain, I've worked on a project recently which was
commited to TDD [not my fault - that's how it was] Now, nobody
on the project knew the first thing about the problem domain.
Yet, we were told that all we have to do is to make the code
pass the tests. We did, tests passed. Do you think the customers
were happy? Well, think again.
What was the problem? Hard to say. It may be that the guys
who wrote the tests didn't have any idea of what they were
testing (or trying to). But, more importantly, the implementors
didn't know either. There were no specs except for the set of
tests. The tests passed, the customer wasn't happy.
At that stage something hit the fan but we could simply point
to the written and signed contract which said all we have to
do is to pass those tests. Well, we had. The system doesn't exist
since it didn't meet the specs (hold on, what specs? There were
no specs...)