nunit-console return codes

[click...@gmail.com]

Would it be possible to change the return codes which nunit-console.exe returns?  We can already distinguish
  0: everything passed
  2: some ignored
  3: some failures
  4: some errors
and can extract the most pertinent information from the resulting XML file accordingly, but inconclusive results also return zero, as do Assert.Ignore(), so these cannot be detected and reported in the same way.

(I'm running 2.6.3 at present, if that makes a difference to your answer)

[Charlie Poole]

I'm not sure what you mean by "we can already distinguish 0.. 1...
2... 3... 4..."

Those are not nunit-console return codes. It returns 0 for no
failures, a positive value equal to the number of test errors and
failures, or a various other negative codes for things for global
errors like file not found, etc.

Changing what we return for NUnit 2.x would be a BBC (Big Breaking
Change) for many folks, which is why we haven't done it. The return
codes we use have been in place for many years.

OTOH, we can certainly rethink the return codes for NUnit 3.0, which
leads me to some questions:

1. Why do you want to distinguish errors and failures? NUnit does make
that distinction individually, since fixing errors and failures often
require a different approach, but in the aggregate, they both mean
that some test didn't work.

2. Why do you care about Inconclusive. The intended definition of
Inconclusive is that nobody cares about the test. That's how we use it
internally with regard to Theories. If you are using Inconclusive
manually in non-Theory tests, you may want to rethink how you are
using it or - alternatively - get us to add a result type that suits
your intention.

I do like the idea that Ignored tests should be highlighted using some
return code. Historically, NUnit has pushed the idea that Ignored
tests are a "Bad Thing" but we don't reflect their presence in the
return code.

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at http://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Mark Levison]

Suggestion modern programs shouldn't rely on return codes to say anything meaningful - instead they should use the output to log real information. The return codes should really just tell the caller something interesting happened go read the XML.

Cheers

Mark

--

Mark Levison | 1 (877) 248-8277 | Twitter | LinkedIn | Facebook Certified ScrumMaster Training: Vancouver | Edmonton | Ottawa | Montreal | Toronto Agile Pain Relief Consulting | Notes from a Tool User Proud Sponsor of Agile Tour Gatineau Ottawa and Agile Coach Camp Canada

[click...@gmail.com]

Ah, my mistake - the test harness I'd used to inspect them had been too limited!
I'll leave you to worry about handling and reporting Ignored tests, but the Inconclusive ones are being used like TODO comments or build warnings - a signal that we ought to do something but it's not critical that we do it right away (that said, we aim for warning-free builds every night, and achieve considerably better than 50% success on it!) so our tests say Assume.That(something we know actually doesn't yet work) so we remember to go back and either fix the code or fix the test.  I've worked around the zero return code by manipulating my xslt slightly, so that's now sorted, but the short answer to your question is "we use Inconclusive as a test warning rather than an error".

Thanks for putting me straight on what the RC actually is.

[Charlie Poole]

That's a very good point. It's impossible to encapsulate even the summary info in the file by a single number.

The main purpose of a return code in this case is to allow a decision as to whether the build is broken. Generally, that test means any non-zero number.

If we return a non-zero code for Ignored tests, which in principle sounds like a good idea, then those who don't want to break the build for Ignored tests will need to check for anything other than zero or (e.g.) one. That's possible in a hand-crafted CI system, but I don't know if it is in the popular CI products - we'll have to check what they support before making a final decision.

I'm inclined to keep our current system but consider adding a command-line option that causes Ignored tests to be treated as errors, at least for the purpose of the return code.

Charlie

[Charlie Poole]

We're considering adding a Warning level to tests, which might solve
your problem while leaving Inconclusive as non-reported except in
special circumstances.

Charlie

[Greg Young]

Returning a nonzero for ignored tests also doesn't make sense in the usage of return codes

Sent from my iPhone

[Charlie Poole]

Hi Greg,

Your brevity has defeated my understanding!

Charlie

[Greg Young]

Return values from processes have specific meaning. A nonzero is a failure. Counting ignored tests As failures breaks the model

Sent from my iPhone

[Charlie Poole]

True... and this may underline that NUnit's calling these tests
"Ignored" was an original error.

In coaching, I have always taught people that Ignored tests were BAD
and should be tracked and removed - not just ignored by the team.
However, IMO, they are not SO bad that the build should be broken by
them... just as you suggest.

I was guessing that clickrick46 wants to see the build fail when tests
are ignored. Is that right click?

To my mind, ignored tests are a sort of warning. We often use a
compiler setting that treats warnings as failures, so it doesn't seem
completely unreasonable to have NUnit do the same thing. I'm not
saying I'm convinced yet, just that it's not something I reject out of
hand.

Charlie

[Greg Young]

If its "treat warnings as errors" no worries but there is a ton of prior art on the meaning of return values :)

Le doute n'est pas une condition agréable, mais la certitude est absurde.