Multiple asserts in a single unit test

[Daniel Berger]

Hi all,

I came across an article regarding unit testing philosophy that I
thought was interesting:

http://weblogs.asp.net/rosherove/archive/2005/04/14/AvoidMultipleAsserts.aspx

This sparked a conversation on IRC regarding coming up with a way to
have asserts continue even if one of them fails within a unit test. Is
there a way to alter the behavior of test-unit so that it doesn't bail
out on subsequent asserts if one fails? Something builtin to test-unit
or a simple hack?

Any opinions on multiple asserts within a single unit test? Any
opinions on what the default behavior should be for test-unit with
regards to multiple asserts where one fails?

Regards,

Dan

[Pit Capitain]

Daniel Berger schrieb:

> I came across an article regarding unit testing philosophy that I
> thought was interesting:
>
> http://weblogs.asp.net/rosherove/archive/2005/04/14/AvoidMultipleAsserts.aspx
>
> This sparked a conversation on IRC regarding coming up with a way to
> have asserts continue even if one of them fails within a unit test. Is
> there a way to alter the behavior of test-unit so that it doesn't bail
> out on subsequent asserts if one fails? Something builtin to test-unit
> or a simple hack?

It may not be simple, but at least it's a hack:

require 'test/unit'

module ErrorCollector

def collecting_errors
is_collecting = @is_collecting
@is_collecting = true
yield
ensure
@is_collecting = is_collecting
end

def raise( * )
super
rescue Test::Unit::AssertionFailedError
handle_error( :add_failure, $! )
rescue StandardError, ScriptError
handle_error( :add_error, $! )
end

def handle_error( method, error )
bck = error.backtrace
bck.shift
if @is_collecting
bck.slice!( 5, 2 )
send( method, error.message, bck )
else
Kernel.raise( error, error.message, error.backtrace )
end
end

end

class SampleTest < Test::Unit::TestCase

include ErrorCollector

def test_multiple_errors
collecting_errors do
assert_equal( 1, 2 )
assert_equal( 2, 3 )
end
assert_equal( 3, 4 )
assert_equal( 4, 5 )
end

end

This is the output:

Loaded suite C:/tmp/r
Started
FFF
Finished in 0.015 seconds.

1) Failure:
test_multiple_errors(SampleTest) [C:/tmp/r.rb:42]:
<1> expected but was
<2>.

2) Failure:
test_multiple_errors(SampleTest) [C:/tmp/r.rb:43]:
<2> expected but was
<3>.

3) Failure:
test_multiple_errors(SampleTest) [C:/tmp/r.rb:45]:
<3> expected but was
<4>.

1 tests, 3 assertions, 3 failures, 0 errors

Regards,
Pit

[Martin DeMello]

Pit Capitain <p...@capitain.de> wrote:
> > out on subsequent asserts if one fails? Something builtin to test-unit
> > or a simple hack?
>
> It may not be simple, but at least it's a hack:

Quote of the week :)

martin

[Phlip]

Daniel Berger wrote:

> Any opinions on multiple asserts within a single unit test?

The distinction is roughly between testing greenfield code with testing
legacy systems.

In greenfield code, test-first can force so much decoupling that you don't
/want/ to assert more than one thing in each case. So setting "one assertion
per case" as a design goal

Some tests stretch the definition of "one assertion":

def test_def_paren
tokenize( "def foo(bar)" )
assert_next_token :keyword, "def "
assert_next_token :method, "foo"
assert_next_token :punct, "("
assert_next_token :ident, "bar"
assert_next_token :punct, ")"
end

That is an assertion of one variable (a hidden @member), chained out to show
each tested detail in the variable. (It's from Jamis Buck's Syntax module.)

Legacy systems, by contrast, typically test at a higher level. If you test a
Tk GUI, simulating a click on a Tk Canvas might change many configurations.
Rather than waste the test state, and the time required to paint a canvas,
write a list of assertions that check each important configuration.

> Any
> opinions on what the default behavior should be for test-unit with
> regards to multiple asserts where one fails?

If you have a debugger, such as for VC++, an assertion should raise a
breakpoint, giving a programmer the option to run more lines.

Unattended, some assertions should drop-dead, and some should keep going.
However, assertion systems already have numerous permutations (assert_equal,
assert_match, assert_nil, modulo _not, etc.). Adding another permutation
would blow our minds.

In C++, I use a raw ASSERT() to drop an entire test run dead, and use
CHECK() for relatively recoverable situations. CHECK(), in unattended mode,
keeps going.

Ultimately, the rule for Test-Driven Development is you don't coddle the
test failure. Your rig should provide automated navigation to the failing
assertion, and should reflect its variables and values. You should either
instantly fix the problem or instantly use Undo to run back to the last
failing state. Coddling the test failures - making a list of them, logging
them, e-mailing them to your boss - are all secondary considerations.

So, all assertions could just drop a test run dead, and TDD will work fine.
Your nightly unattended test run will always pass, so recovering from
assertions is irrelevant.

--
Phlip
http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

[Phlip]

> In greenfield code, test-first can force so much decoupling that you don't
> /want/ to assert more than one thing in each case. So setting "one
assertion
> per case" as a design goal

>sigh<

...tends to decouple.

Friggin' Alzheimers...

> --
> Phlip
> http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

[Douglas Livingstone]

On 4/15/05, Daniel Berger <djbe...@hotmail.com> wrote:

> This sparked a conversation on IRC regarding coming up with a way to
> have asserts continue even if one of them fails within a unit test. Is
> there a way to alter the behavior of test-unit so that it doesn't bail
> out on subsequent asserts if one fails?

IMO, the number of passes plus the number of fails should equal the
number of asserts, regardless of which tests fail. Phlip seems to
agree, but that there should only be one assertatin per test anyway,
so that this iregularity won't be a problem. I find that an artificial
constraint on writing unit tests, so imo tests should continue even
after a failed test.

Also, take this test:

def test_too_true
assert false, "One failed"
end

def test_true
assert false, "Two failed"
assert false, "Three failed"
end

With "break on first fail", this gives:

"One failed"
"Two failed"

Fixing "One" gives:

"Two failed"

Then you fix "Two" and it gets replaced with:

"Three failed"

In the first case, fixing a fail removed it from the list. In the
second case, it was simply replaced with something else, and the only
difference was which test the assertation was put in. It shouldn't
matter where the asserts go, the behaviour of solving something should
be that you stop seeing that it fails, nothing more need happen.

Douglas

[Eric Hodel]

I prefer having multiple tests per method, each testing some other path
through the method. This seems to be a workable alternative (for me!)
to continuing despite failures.

I feel I get the same amount of information regarding potential failure
points that the continue method gives. It also encourages me to write
shorter tests because continuing could lead to tests that are too long.

--
Eric Hodel - drb...@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04