Illegal Argument 45 - Precondition for tests.

[mP]

Firstly a slight nitpick, preconditions is a bit overloaded in this
case given its other uses.. but anyway.

It seems to me that the problem described in the podcast can be best
"solved" by simply grouping ones tests and running the correct group
at the correct time in the appropriate environment. If a certain
resource can only be tested at night in the dark then dont try and run
"that" group in the day time.

Dunno if its just me but what was described seems to be adding
complexity whtn the problem can quite easily be solved in a simpler
way.

[Mark Derricutt]

Possibly, however for my own integration tests, they go through a
series of groups during the integration test.

1) First up the database is dropped/recreated from fresh schema with seed data
2) Next an embedded ApacheDS LDAP server is started and our LDAP
schema loaded with seed LDAP data
3) An OSGi container is started and bundles deployed
4) Main tests run

At any point of 1, 2, 3 failing is potentially a pre-condition fail
for the main tests. Those tests would be red, but the rest (under
TestNG) go yellow for skipped.

--
Pull me down under...

[Richard Vowles]

So segmenting your tests and adding in test block scheduling is simpler than specify the conditions under which a test cannot sensibly run? I'm going to disagree there.... I also see huge value in actually putting the preconditions into a test as it adds extra logical value. In Easyb, we have had requests for being able to fail out in the "given" and "when" phases to do just this. My opinion is that being able to indicate these are still different kinds of failures to the "assertions" is valuable. 

On Fri, Jul 23, 2010 at 1:03 PM, mP <miroslav...@gmail.com> wrote:

--
---
Richard Vowles,
Grails, Groovy, Java
Consistency is the last refuge of the unimaginative - Oscar Wilde
ph: +64275467747, linkedin, skype:rvowles
get 2Gb shared disk space in the cloud - Dropbox, its incredibly useful! - http://tinyurl.com/cmcceh
podcast: http://www.illegalargument.com

[mP]

On Jul 23, 11:35 am, Richard Vowles <rich...@bluetrainsoftware.com>
wrote:

> So segmenting your tests and adding in test block scheduling is simpler than
> specify the conditions under which a test cannot sensibly run? I'm going to
> disagree there.... I also see huge value in actually putting the
> preconditions into a test as it adds extra logical value. In Easyb, we have
> had requests for being able to fail out in the "given" and "when" phases to
> do just this. My opinion is that being able to indicate these are still
> different kinds of failures to the "assertions" is valuable.

This sounds like introducing environment or configuration into the
test. Now your code needs to know that failure is reported as a
"orange" rather than "red" if its a Sunday arvo. What does one do if
the environment or rules when the test can fail change ? Given that
this test says it can fail, does that mean that if this test fails, we
just ignore it is a permanent answer to the problem. What if we have
set up the environment or some requirement incorrectly and this means
the precondition *always* fails even though it should pass some of the
time. I would say introducing this fuzzy determinism means the test is
never a red fail because if something is wrong we choose to report it
as an orange and nevere a red. What if we want a failure to now be a
red because we want the definitive answer to all tests in all forms
now ? How do we force this, do we need to change the precondition code
somehow ?

Maybe i dont get it, but it seems a hell of a lot simpler to have red/
green outcomes and running the right test at the right time in the
right environment ? It seems a lot easier to externalise the when and
what tests are run outside the actual java Tests than inside them.

[Mark Derricutt]

Re-replying to the list - not sure how this became a private conversation:

Well, looking at a few "test failures" we've had in the past:

- PostgreSQL was unable to drop/recreate the database due to a
lingering open transaction from an earlier aborted test run. This
means having to log in and run "rollback prepared '${transactionid}'".
Doesn't happen often, but has happened a few times.
- Database Server is out of disk space so unable to create db -
again, doesn't happen often but has happened.
- Problems elsewhere in the network caused the iSCSI filesystem used
by the build server to become READ-ONLY - so also unable to create
databases

In these cases the "tests" never ran, so is definitely an
environmental precondition failure.

--
Pull me down under...

On Sat, Jul 24, 2010 at 12:52 PM, mP <miroslav...@gmail.com> wrote:
> Im not sure, if your response is a good example that backs the need
> for preconditions. Most (all?) of the 4 steps that you describe should
> always work if one assumes that the db is some local (im memory etc).
> I would say that the above should always work but introducing some
> precondition to just ignore them because its tricky or hard is a bad
> way to write reliable reproducable tests.
>

[mP]

No i hit the wrong "reply" by mistake.

Even if the DB ran out of space, it still makes sense IMHO to mark
that test as a fail. What happens if your new stuff needs to be tested
w/ the real DB ? If your going to ignore results when stuff goes wrong
w/ a series of tests you might as well delete them.

[Richard Vowles]

There is no intention to "ignore" here. The whole point is to give more information that you otherwise get. A fail means the test ran and gave the wrong results. A precondition fail means we couldn't run the test and any result is not meaningful. 

JUniti already has this: http://junit.sourceforge.net/javadoc/org/junit/Assume.html

It just gives what I would consider the wrong result. 

[Richard Vowles]

And hilariously enough, it uses Mark's exact example.

[mP]

On Jul 24, 2:28 pm, Richard Vowles <rich...@bluetrainsoftware.com>
wrote:

> There is no intention to "ignore" here. The whole point is to give more
> information that you otherwise get. A fail means the test ran and gave the
> wrong results. A precondition fail means we couldn't run the test and any
> result is not meaningful.

If your system threw nice exception messages, why would your tests
need to his extra detail ? Shouldnt your components throw nice
messages themselves to centralise ? If they do then most of th etime
this use of assume is not necessary. So oh the test x does y fails oh
i can see the stack trace says something about the whatever not being
available.

>
> JUniti already has this:http://junit.sourceforge.net/javadoc/org/junit/Assume.html
>

The javadoc for Assume says

-> A failed assumption does not mean the code is broken, but that the
test provides no useful information. The default JUnit runner treats
tests with failing assumptions as ignored. <-

The default says the test is ignored, im confused how this makes
anything better.

How does that help anyone solve the problem of running some test
reliably, so that it works or it fails if something is wrong ? Yes
integration testing can be a pain because of outside factors, but
ignoring a test because the envirnoment was screwed is not the answer.
Seeing as the default is to ignore, most of the time you will not be
aware that something was ignored/skipped whatever.. which again means
you are getting false impression that everything and i mean ALL tests
passed.

> It just gives what I would consider the wrong result.
>

[mP]

I am just commenting that one can get away with just the plain old
assert system. There is no need for multiple "types" of assert it just
makes life complicated. Too much choice is also a bad thing. Take a
look at j.u.l. with its logging levels.. Which exactly does one use to
log debug messages. Is there a difference ? If i want to see debug
stuff which level do i use ? Is there a need to differentiate. No keep
it simple - one is enough...

[Nigel Charman]

Ignoring the example in the Javadocs, I believe the real intention of
Assume is as a quantifier, or filter, for JUnit Theories. For example:

@Theory validPasswordsAreAccepted(String password) {
assumeTrue(password.matches(PWD_REGEX));
assertTrue(passwordChecker.isValid(password));
}

Prior to the demise of Agitar, David Saff described a new Theory
Explorer tool[1] that would search for new inputs that passed the
assumptions and exercise the theory. This tool never came to light.

I've never seen any recommendations to use Assume for pre-condition
failure.

nigel.

[1] See http://groups.csail.mit.edu/pag/pubs/test-theory-demo-oopsla2007.pdf

On Jul 24, 4:28 pm, Richard Vowles <rich...@bluetrainsoftware.com>
wrote:

[Nigel Charman]

This discussion has prompted me to finish the blog post I started
after CITCON. I'm arguing that we should differentiate test failures
on the grounds that different errors may need to be assigned to
different roles (eg. data failures to testers, system failures to sys
ops, code failures to devs), and for reporting purposes. See
http://tutansblog.blogspot.com/2010/06/triaging-test-failures.html.

[mP]

So whats the concensus ?

4 different people, 3 different comments :)

[Richard Vowles]

I don't think there was a consensus was there?

On Tue, Aug 3, 2010 at 4:45 PM, mP <miroslav...@gmail.com> wrote:

So whats the concensus ?

4 different people, 3 different comments :)

[Nigel Charman]

I think we can all agree not to use Assume in JUnit :-) (unless using
JUnit Theories). Mark's consent is automatic, since he wouldn't touch
JUnit with a barge pole.

On categorising test failures, we beg to differ.

On Aug 3, 5:01 pm, Richard Vowles <rich...@bluetrainsoftware.com>
wrote:

> I don't think there was a consensus was there?
>

> On Tue, Aug 3, 2010 at 4:45 PM, mP <miroslav.poko...@gmail.com> wrote:
> > So whats the concensus ?
>
> > 4 different people, 3 different comments :)
>
> --
> ---
> Richard Vowles,
> Grails, Groovy, Java
> Consistency is the last refuge of the unimaginative - Oscar Wilde
> ph: +64275467747, linkedin, skype:rvowles

[mP]

Im glad you kind of agree w/ me, that assume is unnecessary and
regular asserts are enough. Im only glad because maybe this feature is
too controversial and means many things to many people which causees
inconsistency and confusion.

[Mark Derricutt]

Not a barge pole no - maybe a baseball bat tho ;)

However, the more I look at Infinitest I get tempted.

--
Pull me down under...

[Mark Derricutt]

It's my thought that the finer grained reporting you CAN get the better, whether you make use of it depends on the project and problem space.

After all - Exception is good enough to be thrown for all errors right?    IllegalArgumentException, IOException, FileNotFoundException are all rather "controversial, mean many things to many people, and cause inconsistency and confusion" - but we need that differentiation of exceptions to separate out a chain of responsibility.

We know this in our code, but I think we also need this for our builds/tests.

--
Pull me down under...

[Richard Vowles]

He didn't agree with that, he agreed that the use of the assumeThat in JUnit without the use of Theories was not useful. He still thinks there should be an precondition test.

On Thu, Aug 5, 2010 at 4:10 PM, mP <miroslav...@gmail.com> wrote:

--
---
Richard Vowles,
Grails, Groovy, Java
Consistency is the last refuge of the unimaginative - Oscar Wilde
ph: +64275467747, linkedin, skype:rvowles

[mP]

Not quite most of the time one knows what each of those exceptions
represents. Lets think functional for a moment and use a simple method
as a function. In this case of a simple function there are two
possible outcomes pass (a value was returned ) or failure. An
exception heirarchy is a way of categorising any sort of problems.
Checked exceptions one could argue are an interesting way to limit
this graph of potential problems. However it becomes tricky to cope
with the difference with runtime vs checked one some exceptions are
placed in the wrong bucked.

Assume is just like the problems in API design with runtime/checked
exceptions. Using the wrong thing at the wrong time, causes ugliess
and silly workarounds. Somethings should nenver be runtime exceptions,
checked exceptions have their place and should be dealt with rather
than letting all exception escape to some big catch up there.

IMHO assume is nasty because, how does one test that an environment is
every correct ? Any time the environment fails, it wont be reported as
an assertion failure. Now a simple report has more than 2 totals.
Tests either pass or fail. If you dont want a test to run because the
environment is "wrong" etc dont run it. Your reports will only have 2
numbers and alls well. If you know your environment ( lets face it the
environment is the biggest pest in this case) is now right and ready
to go, run those environmentally sensitive tests..

[mP]

This categorisation via assume makes things more complicated as
outcomes now have more than just 2 numbers, passes and fails. It also
wastes and can potentially add considerable time to any run because
one is running tests that will always fail. Running no code is always
going to be quicker than some precondition test no matter how quick
that test is.

[Richard Vowles]

I can not only cope with more than two totals from my tests, I can gather valuable information from that. 

[Nigel Charman]

i like your line of thought :-)

On Aug 5, 4:18 pm, Mark Derricutt <m...@talios.com> wrote:
> It's my thought that the finer grained reporting you CAN get the better,
> whether you make use of it depends on the project and problem space.
>
> After all - Exception is good enough to be thrown for all errors right?
> IllegalArgumentException, IOException, FileNotFoundException are all rather
> "controversial, mean many things to many people, and cause inconsistency and
> confusion" - but we need that differentiation of exceptions to separate out
> a chain of responsibility.
>
> We know this in our code, but I think we also need this for our
> builds/tests.
>
> --
> Pull me down under...
>

[Nigel Charman]

Let's park "assume" since none of us are advocating it.

Richard and I both have use cases where we could benefit from finer
grained outcomes being reported. That's not incompatible with your
use case where you just want red/green. In Mark's terminology, you'd
just catch Exception, whereas Richard and I might catch subclasses.

nigel.

PS. Unit tests already have at least 4 potential outcomes - compile
error, error (unexpected exception), fail, pass.

[mP]

I prefer to think of test suites like a java control statements if/
while/do etc in that they only take booleans with no auto conversion
nonsense. Its all true/false for me, no tenary t/f/null etc.