Why must test methods return void?

[Charlie Poole]

Hi All,

Here's a question that led to an interesting discussion on the
JUnit list...

> After over 10 years of using JUnit, it has only just struck me that test
> methods must return void, but why?
>
> I've just come across an example where a particular test did a lot of
work,
> and I wanted to piggy-back on its final state by calling the method from
> another test. Is there any fundamental reason for not allowing me to
> return values? It might even be an opportunity to write an annotation
> expressing constraints about the value returned.

The discussion pointed out some alternative ways of dealing with the
problem - I won't go into them here - and led to this comment by Kent:

> The initial design was a historical artifact. The little code snippets
> I used as tests didn't have a return value, so the tests didn't have a
> return value. I never found a compelling reason to add a return value,
> although I've seen frameworks that relied heavily on return values from
> tests.
>
> What I wonder is if we should relax the constraint that tests cannot
> return a value. In the olden days it made sense because you didn't want
> someone to just happen to name a method "test..." and have it accidentally

> run as a test. With @Test, this risk disappears.

We could say the same about NUnit... Requiring a test to return void
is an artifact of NUnit 1.x days. With 2.x, we might have removed the
restriction but did not.

When we introduced parameterized tests, we allowed a return value,
but only if the attribute or data source specified an expected return
value to be tested by NUnit.

All of these restrictions come from an attitude of protecting the user
from mistakes - a well-meant intention, but perhaps not needed so much
any longer.

What if we simply removed the restriction for *all* tests and simply
ignored the return value if no 'Result=' was specified?

Charlie

[Simone Busoli]

And what would you use the return value for? Would you really like to promote calling a test from another test?

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To post to this group, send email to nunit-...@googlegroups.com.
To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.

[Charlie Poole]

Hi Simone,

 

Well, by definition, NUnit wouldn't use it for anything. Currently we only *permit* it

if there is an annotation that tells us what to do with it.

 

OTOH, the user might write a utility method in the form of a test and then reuse

it in another test. For example, take this code...

 

[Test]

public void CanCreateNewComplexObject()

{

    ComplexObject co = CreateNewComplexObject();

    // Assert about the object

}

 

[Test]

public void CanCreateNewComplexObjectAndDoSomethingWithIt()

{

   ComplexObject co = CreateNewComplexObject()

   co.DoSomething();

   .// Asserts about the result

}

 

Could be re-written as...

 

[Test]

public ComplexObject CreateNewComplexObject()

{

   // Create the object

   // Asserts

   return theObject;

}

 

[Test]

public void CanCreateNewComplexObjectAndDoSomethingWithIt()

{

   ComplexObject co = CreateNewComplexObject()

   co.DoSomething();

   // Asserts about the result

}

 

Pros:

* One less method

* Basic asserts are run each time

 

Cons:

* Possibly less clear

* Basic asserts are run each time. :-)

* Used inappropriately, could lead to creation of fragile networks of interdependent tests

 

 

A bit of historical perspective: NUnit (like JUnit) started out with an 'attitude' that

there would be no features that might lead naive users to use it inappropriately.

I believe users have become more sophisticated and I hope that I - as the

developer - have become more humble about telling people how to program.

 

However, I don't have a strong position either way on this, I'm just asking.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Simone Busoli
Sent: Friday, January 22, 2010 1:47 PM
To: nunit-...@googlegroups.com
Subject: Re: [nunit-discuss] Why must test methods return void?

[Simone Busoli]

Well, I think that you can provide somewhat dangerous features as long as they are useful in certain circumstances, and provide means of accomplishing something which can hardly be accomplished otherwise, but here we're really talking about saving one method, which can be easily worked around by applying an extract method refactoring.

I cannot think of a use case where it might be much complex than this, to justify letting call a test from another test, is it just me?

[Charlie Poole]

Hi Simone,

 

I can't help but give some weight to the fact that Kent didn't

think it was absolutely crazy. I suspect that he - like me - was

surprised to realize that his software placed an unintended

restriction on the user.

 

Note - we aren't talking about adding a feature here. The 'feature'

we already have is that we check the return and won't allow the

user to use a method returning a value except in specific

circumstances.

 

But - like I said - I have no dog in this fight. :-)

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Simone Busoli

Sent: Friday, January 22, 2010 2:23 PM

[Adam Connelly]

On a slight aside, I think that NUnit could do with better documentation. We should be recording the outcome of discussions like this somewhere that's easy to find. I think if this was the case, you'd have to worry less about "naive" users making mistakes.

It could be good to have:

 * General documentation for NUnit.

 * Quick reference for the different asserts (might be covered by the above).

 * Techniques for testing various scenarios.

 * Some kind of section on stuff like "should I test private methods", etc.

Sorry for kind of hijacking this discussion, but I thought it was kind of related.

Cheers,

Adam

[Charlie Poole]

Hi Adam,

 

You bring up some good points.

 

With regard to "recording the outcome" ... I think that's a separate issue from documentation.

It would be very confusing to mix what we plan to do (and which could always change) with

what actually is.

 

Nevertheless, it's something I've been thinking about. In fact, so much so that last night I started

going through the archives to see how well our discussions were being followed up. I'm not done

yet, but it isn't too bad. Generally, one or more of several things happen when a discussion

comes to a conclusion...

 

1) I've written some notes to the list with a subject like "xxxxxx - Resolved" saying what the

conclusion is.

 

2) Either I or someone else will file a feature request.

 

3) Less often, either I or someone else writes a specification on the wiki, which may or

may not be tied to a Launchpad blueprint

 

4) The decision slips into the code (and docs for the next release)

 

I'm not done reviewing yet, but I think it's likely that any confusion you sense around the

lack of a clear outcome is because there has not been one yet. :-)

 

Regarding the docs themselves, I think we have only the "General documentation" and

there is certainly room for improvement there. Any specific ideas?

 

Answers to "should I test private methods" belong in the general FAQ, in my view. Feel

free to get an item started and let us know so we can all chime in. For a question like

that one, it could be an interesting process.

 

I'm not sure where "techniques for testing various scenarios" belong. We could certainly

put short items about techniques in the FAQ, but more extended treatment probably

belongs in a book - like the one I keep trying to find time for these days. :-)

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Adam Connelly
Sent: Saturday, January 23, 2010 2:33 AM

[gerry_lowry (alliston ontario canada (705) 250-0112)]

This is the view of a unit testing neophyte:

Tests need to be standalone.

Example:

[Test]
public void MyStarTrekDateCalculator
{
// Arrange ...
// Act ...
Assert.AreEqual(7.5,daysBetweenStarDates(1313, 1320.5);
Assert.IsFalse(False, IsValidStarDate(-02.0342156)
}

The above example is poor imo because failure is not really clear; i.e.,
it's not a true standalone test because of the multiple asserts and also
because the test name is not clear, and lastly because if the first assert
fails, AFAIK, the second assert would not be tested.

Revision:

[Test]
public void MyStarTrekDateCalculatorShouldComputeDaysBetweenStarDates
{
// Arrange ...
// Act ...
Assert.AreEqual(7.5,daysBetweenStarDates(1313, 1320.5);
}

[Test]
public void MyStarTrekDateCalculatorShouldDetectInvalidStarDate
{
// Arrange ...
// Act ...
Assert.IsFalse(False, IsValidStarDate(-02.0342156)
}

Now, above we have two standalone independent unit tests.
The return type is void because the test runner will tell me
whether they pass or fail.

Speculation/Philosophy
================
A non void return type might be useful ONLY for communication with
the test runner; never for inter-test communication.

In the first case, the communication might be an int return code
where zero is the test ran and non-zero might mean that something
broke in the test framework and therefore the test can not be
properly evaluated.

In the second case, inter-test communication leads to dependencies.
This has a high probability of the tests being fragile tests,
Also, should NUnit add method order random execution, as
currently found only in xUnit.net AFAIK, then such interdependent tests
would surely disentigrate.

Gerry

[Charlie Poole]

Hi Gerry,

> This is the view of a unit testing neophyte:
>
> Tests need to be standalone.

I'd say *unit* tests need to be standalone - at least as we
define unit tests (microtests?) these days.

It's important to remember that folks are using NUnit for
other kinds of tests.

I think you misunderstood the idea behind the request. It
wasn't at all to do with inter-test communication but with
allowing a method to be *both* a utility method and a test
method without introducing an extra level of indirection.

However, if you thought of this, then others may do the same
and use it as a way of communicating between tests. That's
a bad idea in my book.

> In the first case, the communication might be an int return
> code where zero is the test ran and non-zero might mean that
> something broke in the test framework and therefore the test
> can not be properly evaluated.

All these bases are currently covered by NUnit, without needing
to rely on the test-writer to do it. I think that's preferable.

> In the second case, inter-test communication leads to dependencies.
> This has a high probability of the tests being fragile tests,
> Also, should NUnit add method order random execution, as
> currently found only in xUnit.net AFAIK, then such
> interdependent tests would surely disentigrate.

Again, inter-test communication wasn't the intent here. However,
I can imagine someone looking at the notion and then trying to
use it that way. That's definitely a con.

I'm seeing a lot of potential confusion around this issue, which
may be a good reason not to introduce it. Aside from ordering,
I see some possible confusion for tests that currently return
a value to be tested by NUnit. If they are called directly,
NUnit won't be in control to make that test. This can be
explained in the docs, but it's likely to confuse people.

I'm leaning toward keeping the current restriction.

Charlie

>
> Gerry

[Simone Busoli]

I agree for this exact reason. I think it would be misused, and since it wouldn't provide anything that cannot be achieved by a simple refactoring, I personally wouldn't introduce it.

[Yann]

Hi Gerry,

I don't profess to be a unit testing expert, but it seems to me that you're
mixing your "act" & "assert" into just "act".

Personally I would have written your test like this:

[Test]
public void MyStarTrekDateCalculatorShouldComputeDaysBetweenStarDates
{
// Arrange ...

// Act ...
int result = daysBetweenStarDates(1313, 1320.5)

//Assert ...
Assert.AreEqual(7.5, result);
}

Hope that's helpful, but happy to be "corrected" by anyone who thinks
otherwise.

Also, as a matter of personal style, I would make a couple of additional
changes:

[Test]
public void MyStarTrekDateCalculatorShouldComputeDaysBetweenStarDates
{
// Arrange ...

int expected = 7.5

// Act ...
int result = daysBetweenStarDates(1313, 1320.5)

//Assert ...
Assert.That(result, IsEqualTo(expected));
}

To me that "reads" a little better.

Yann

--------------------------------------------------
From: "gerry_lowry (alliston ontario canada (705) 250-0112)"
<gerry...@abilitybusinesscomputerservices.com>
Sent: Sunday, January 24, 2010 6:15 AM

To: <nunit-...@googlegroups.com>
Subject: Re: [nunit-discuss] Why must test methods return void?

> This is the view of a unit testing neophyte:

[gerry_lowry (alliston ontario canada (705) 250-0112)]

Hi Yann,

you are correct; in this case, I deliberately wrote the examples as I did because
I wanted to focus on the subject of this e-mail.

that said, as I am neophyte with regards to creating canonical unit tests,
I sometimes find the lines between arrange--act--assert a bit blurry.

Your example has clarified that for me somewhat and led me to
visit Google where I'm finding some very useful links, for example,
<http://agileinaflash.blogspot.com/2009/03/arrange-act-assert.html>,
<http://www.dotnetnuke.com/Community/Blogs/tabid/825/EntryId/2466/Adventures-in-Testing-1-AAA-Grade-Quality-using-Arrange-Act-Assert.aspx>,
et cetera; more at Google, example: [explain arrange act assert unit test].

I'm finding there are some interesting interpretational fluctuations with regards
to expressing AAA in code, even among unit testing experts.

Gerry

[Yann]

Hi Gerry,

Yes, I did wonder whether you had done it on purpose, in the end the fact
that you'd left the 3 AAA comments (& that you claimed to be a neophyte)
finally made me think that perhaps you hadn't.

I wasn't in any way suggesting that "my way" is the "right way" (I hope I
didn't give that impression), but it's the way I've found most helpful for
my own test coding.

There are *always* different interpretations from the "experts". I try to
take from what they say the things that help me do what I need to do, rather
than rigorously adhering to any one philosophy. Far too many people seem to
get bogged down in the dogma.

Anyway, I'm glad it was perhaps mildly helpful & happy to see it was food
for further thought & googling. :-)

And thanks for the links. I apologise to all if this has wandered too far
from the original topic.

Yann

--------------------------------------------------
From: "gerry_lowry (alliston ontario canada (705) 250-0112)"
<gerry...@abilitybusinesscomputerservices.com>

Sent: Sunday, January 24, 2010 1:59 PM

[gerry_lowry (alliston ontario canada (705) 250-0112)]

Hi again, Yann,

for me, not a problem. regarding right way/wrong way,
I frequently plagiarize TIMTOWTDI from the Perl community:
There is More Than One Way To Do It
is applicable to much of the code that gets written.

Getting back to the original topic, my misinterpretation
of the original topic led Charlie to solidify his own
thoughts ...

in my own experience, I often learn more when teaching
because of the feedback from those whom I've caused
to become perplexed.

also, I agree with Adam Connelly's comments about the need for
better documentation ... not only for NUnit, also for xUnit.net, et cetera.

Gerry

[snip]

[Adam Connelly]

Sorry, I shouldn't have said recording the outcome of "discussions like this". I was more concerned about discussions that are more theoretical (i.e. can't be made into feature requests), because I think I've seen a few of those now. I just feel that someone should be able to go to nunit.org, download NUnit, and then access resources from the site that help them start with TDD and answer as many of the questions that you have when you're just starting as possible.

Sorry to hijack this thread once again.

Adam

2010/1/24 gerry_lowry (alliston ontario canada (705) 250-0112) <gerry...@abilitybusinesscomputerservices.com>