Has NUnit3 changed how Assertion Failures are communicated back to the Runner?

[Gerard Meszaros]

I have a custom assertion. I have  a test for it. I use the following:

[Test]

public void testAssertEquals_FlightDtosNotEqual()

{

FlightDto flightDto = CreateAnonymousFlightDto();

FlightDto differentFlightDto = CreateAnonymousFlightDto();

try

{

AreEqual(flightDto, differentFlightDto, "2 FlightDto should not be equal ");

}

catch (AssertionException)

{

   return;  // Success

}

                       Assert.Fail("Failed to distinguish 2 different FlightDtos");

                }

This test fails even though AreEqual fails. it calls Assert.AreEqual on the field FlightNumber which fails:

               Assert.AreEqual(expected.FlightNumber, actual.FlightNumber, message + " flightNumber");

The failure message is:

" 2 FlightDto should not be equal  flightNumber

  Expected: 1

  But was:  2"

Which is what should happen if I call this method directly from the normal test. But the catch(AssertionException) should invert this.

I have single-stepped through this code and it does go into the Catch clause and then it returns. So the Assert.Fail at the end is never hit. So the test should be considered PASSED. But the test is still considered to have FAILED.  Why?

SO, Has NUnit 3 changed how assertion failures are communicated between the place the failure is detected and the runner?  Is the failed assertion being recorded when it is executed rather than when the exception reaches the runner? 

Or am I missing something else?

BTW, VbUnit used to have this same issue 15 years ago.

Thanks,

Gerard

[Charlie Poole]

Hi Gerard,

This __has__ changed and more change is likely in the future.

To put it in perspective, the use of an AssertionException to communicate failure is an implementation detail. However, it has been in use for so long that many people came to rely on it. The immediate cause of this change is the introduction of warning results and multiple assertion failures per test.

Here's what now happens: When a warning (Warn.If, Warn.Unless, Assert.Warn) or any assertion failure occurs, the information is stored in the pending test result. In the case of a warning, nothing more happens until the end of the test. For true assertion failures, it depends on whether we are inside a Multiple.Assert block or not. If not, then an AssertionException is thrown. If we are in a block, then execution continues till we exit all nested blocks. At that point, if there have been assertion failures the exception is thrown.

When a test completes, either normally or through an exception, NUnit examines the saved failures and sets the test result according to the "worst" outcome, in this order: Warning, Failure, Error. It also constructs a composite failure message, listing all failures for use by runners that are not aware of the new structure of a test result. One side effect of this is that you will only see stack traces for multiple failures if you are using a runner that understands the new structure. The latest console runner does understand the structure and so does the latest NUnit 3 VS adapter - although what it can display is limited by what Visual Studio can understand.

IME, the cases where catching the exception is desirable are really very limited. We have to do it in testing NUnit, of course, but most other cases that have come up in past discussions are easily handled in other ways that don't depend on the internal implementation of NUnit.

For the rare case where it is needed, there's a workaround that we use internally and in some of our tests of NUnit:

    using(new TestExecutionContext.IsolatedContext())
    {

        // your try-catch goes here

    }

This causes the "spoiled" result to be discarded upon exit from the block, so you get a fresh start. Obviously, you shouldn't do any asserts in the block that you _don't_ want to have discarded.

TestExecutionContext is in the NUnit.Framework.Internal namespace. I gave some thought to exposing this as a feature in TestContext, but I really couldn't think of a use case to justify it. If you know of one that's not about testing NUnit, we could do something like

    Assert.Isolated(() =>
    {

        // your code

    });

Alternatively, we could expose something similar in TestContext.

Hope this is helpful!

Charlie

PS: As we move to future releases, we will probably be recording other things in the same way. IgnoreException, SuccessException and InconclusiveException could be similarly affected. Eventually, they might be replaced by a general exception that does nothing except terminate the test.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

[Gerard Meszaros]

On Friday, February 3, 2017 at 12:07:53 PM UTC-7, charlie wrote:

Hi Gerard,

This __has__ changed and more change is likely in the future.

To put it in perspective, the use of an AssertionException to communicate failure is an implementation detail.

I think this is a debatable point. This is externally visible behaviour. That isn't an "implementation detail". 

However, it has been in use for so long that many people came to rely on it.

 In every member of the xUnit family that i have ever used since VbUnit, this was the behaviour. No side-effects. So if you want to check that a Custom Assertion fails when it should, you simply call the assertion inside a TryCatch block (or an Assert.Throws. ) And the result of the test is entirely up to you. There are no "hidden side effects". Changing this is a BIG DEAL.

The immediate cause of this change is the introduction of warning results and multiple assertion failures per test.

When was this breaking change introduced? I trolled through the list of breaking changes and didn't see it.

 

Here's what now happens: When a warning (Warn.If, Warn.Unless, Assert.Warn) or any assertion failure occurs, the information is stored in the pending test result. In the case of a warning, nothing more happens until the end of the test. For true assertion failures, it depends on whether we are inside a Multiple.Assert block or not. If not, then an AssertionException is thrown. If we are in a block, then execution continues till we exit all nested blocks. At that point, if there have been assertion failures the exception is thrown.

When a test completes, either normally or through an exception, NUnit examines the saved failures and sets the test result according to the "worst" outcome, in this order: Warning, Failure, Error. It also constructs a composite failure message, listing all failures for use by runners that are not aware of the new structure of a test result. One side effect of this is that you will only see stack traces for multiple failures if you are using a runner that understands the new structure. The latest console runner does understand the structure and so does the latest NUnit 3 VS adapter - although what it can display is limited by what Visual Studio can understand.

IME, the cases where catching the exception is desirable are really very limited. We have to do it in testing NUnit, of course, but most other cases that have come up in past discussions are easily handled in other ways that don't depend on the internal implementation of NUnit.

For the rare case where it is needed, there's a workaround that we use internally and in some of our tests of NUnit:

    using(new TestExecutionContext.IsolatedContext())
    {

        // your try-catch goes here

    }

This causes the "spoiled" result to be discarded upon exit from the block, so you get a fresh start. Obviously, you shouldn't do any asserts in the block that you _don't_ want to have discarded.

Is this available to users of the framework (i.e. non-internal uses)?

 

TestExecutionContext is in the NUnit.Framework.Internal namespace. I gave some thought to exposing this as a feature in TestContext, but I really couldn't think of a use case to justify it. If you know of one that's not about testing NUnit, we could do something like

    Assert.Isolated(() =>
    {

        // your code

    });

Alternatively, we could expose something similar in TestContext.

Both of these are ugly solutions to a self-imposed problem. It changes something that was trivially easy to test into something that is hard to test. (Assertions were easy to test because they had no side effects.)

Testing custom assertions is the use case. It has nothing to do with testing NUnit itself so it should not have to rely on "internal" features.It as about writing clean tests by building up a DSL of custom assertions that use domain-specific language to express the expectations. (They don't even need to be called "assertions" but they do need to throw AssertionException to fail the test and typically do so by calling framework-provided assertions.) Having to create isolated TestContexts to test a CustomAssertion is relying on "implementation details".

Personally, I think think this is a bad design. One that is right up there with the sharing of the testcase instance among test methods.This will be a stumbling block for people upgrading from earlier versions of NUnit. And it moves NUnit farther and farther from the common feature set of the xUnit family. Pretty soon it will be just as far out as TestNG, the "bastard child" of the family.

   At least make the default behaviour the same as before and force users to enable the new multiple-failure behaviour.  Please!

Hope this is helpful!

It has been very helpful in confirming I wasn't going crazy! The last time I was this confused by any member of the xUnit family was when I tried to implement Lazy Setup in NUnit 2.0 and it didn't work. I spent hours in the debugger before convincing myself that I was doing things right and it was the framework that was leaking information between test methods.  (I can't remember; has this been fixed in NUnit 3? Last I heard you were concern about breaking legacy tests that depended on it. ;-)   ) 

So what can I do right now to let me test my Custom Assertions. I teach this stuff to people in every course I run and I need to be able to tell them how to do it in NUnit 3.x; it wouldn't look very good to have NUnit 3 be the only member of the family in which we cannot test our Custom Assertions. (I already have to explain some of the other "foibles" of NUnit; I'd like the list of exceptions to get smaller, not larger!)  I'm embarrassed to say that I'm still teaching based on NUnit 2.x (and running my exercises in 2.2.8) because of the inability to (meta)test my tests due to the changes in the framework after 2.2.8.

 

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

[Charlie Poole]

Hi Gerard,

Replies inline...

On Fri, Feb 3, 2017 at 10:25 PM, Gerard Meszaros <gmka...@gmail.com> wrote:
>
>
> On Friday, February 3, 2017 at 12:07:53 PM UTC-7, charlie wrote:
>>
>> Hi Gerard,
>>
>> This __has__ changed and more change is likely in the future.
>>
>> To put it in perspective, the use of an AssertionException to communicate
>> failure is an implementation detail.
>
>
> I think this is a debatable point. This is externally visible behaviour.
> That isn't an "implementation detail".

I think we have a fundamental disagreement about the difference
between interface and implementation. Like any software, NUnit
guarantees certain behaviors, but it has many more "observable
behaviors". If anything that is observed is viewed as an unchangeable
part of the interface, then it becomes virtually impossible to change
anything. We - like other software - try to push back against that
tendency by clearly telling people what is guaranteed and what is not.

Of course, if enough people start to rely on something being
implemented in a certain way, you can end up being forced to maintain
it in the future. It's a sort of retroactive feature creep.
Fortunately, I don't think that's the case here, since the impact is
fairly limited.

>> However, it has been in use for so long that many people came to rely on
>> it.
>
>
> In every member of the xUnit family that i have ever used since VbUnit,
> this was the behaviour. No side-effects. So if you want to check that a
> Custom Assertion fails when it should, you simply call the assertion inside
> a TryCatch block (or an Assert.Throws. ) And the result of the test is
> entirely up to you. There are no "hidden side effects". Changing this is a
> BIG DEAL.

I"m sure you aren't trying to say that NUnit is required to work
internally the same as VbUnit, et al. I realize that the difference
could be confusing for someone trying todo similar things in multiple
frameworks, but I don't think that's the case for most users.

Reporting the result of a failure isn't a side effect. It is the
effect of any assertion. In fact, I could just as well call the
throwing of an exception the side effect. The object of running a test
is to return a result after all, not to throw an exception.

>> The immediate cause of this change is the introduction of warning results
>> and multiple assertion failures per test.
>
>
> When was this breaking change introduced? I trolled through the list of
> breaking changes and didn't see it.

It was in 3.6, released a few weeks ago. I didn't put it in the
breaking changes page because I actually wasn't thinking of it as a
breaking change but as a change to an implementation detail. However,
since it can break some existing tests, I should add it there.

Well, you're a user and I just told you how to use it, so... yes!

The Internal namespace exists to tell people that they are using part
of the NUnit implementation, without actually declaring the class as
internal and making it inaccessible to them. People writing
extensions, for example, need to use Types in that namespace.

>>
>>
>> TestExecutionContext is in the NUnit.Framework.Internal namespace. I gave
>> some thought to exposing this as a feature in TestContext, but I really
>> couldn't think of a use case to justify it. If you know of one that's not
>> about testing NUnit, we could do something like
>>
>> Assert.Isolated(() =>
>> {
>> // your code
>> });
>>
>> Alternatively, we could expose something similar in TestContext.
>
>
> Both of these are ugly solutions to a self-imposed problem. It changes
> something that was trivially easy to test into something that is hard to
> test. (Assertions were easy to test because they had no side effects.)

With respect, I think you are focused so much on the problem you are
trying to solve - which is not a common problem for general users -
that you are n
ot paying attention to why it was necessary to do this in the first place.

In C#, there is no ready way to throw an exception and then resume the
same code. That's what would be necessary in order to allow multiple
assertions to be reported using exceptions.

As I explained, the above was a possible way to expose this feature,
which I did not implement. I don't like it either. I'd be happy to
have a suggestion, if you can put yourself in the mood to offer
something positive.

> Testing custom assertions is the use case. It has nothing to do with testing
> NUnit itself so it should not have to rely on "internal" features.It as
> about writing clean tests by building up a DSL of custom assertions that use
> domain-specific language to express the expectations. (They don't even need
> to be called "assertions" but they do need to throw AssertionException to
> fail the test and typically do so by calling framework-provided assertions.)
> Having to create isolated TestContexts to test a CustomAssertion is relying
> on "implementation details".

Personally, I don't think it is possible to write custom assertions
without relying on implementation details. Someone extending the
framework has to understand internals except for the simplest cases.

When you use a try / catch to test, you are using an internal
implementation detail. I've been telling people that for years in our
forums. It's the main reason that user code should not attempt to
catch any exceptions except the ones the tests or the system under
test is known to throw. I've often told people "the next version of
NUnit may not throw an exception." Well, now it has happened!

> Personally, I think think this is a bad design. One that is right up there
> with the sharing of the testcase instance among test methods.This will be a
> stumbling block for people upgrading from earlier versions of NUnit. And it
> moves NUnit farther and farther from the common feature set of the xUnit
> family. Pretty soon it will be just as far out as TestNG, the "bastard
> child" of the family.
>
> At least make the default behaviour the same as before and force users to
> enable the new multiple-failure behaviour. Please!

We'll ignore the insults to the design and take the suggestion into
account. Frankly, I haven't seen the problems you predict so far. One
user with a very obscure use of DelayedConstraint in combination with
Throws.Nothing ran into a problem that forced the use of the isolated
context.

As far as requiring a switch to enable the full features of NUnit,
that seems like a non-starter to me. Most users just want to run tests
and see the results. It's hard to explain why we have added new
features but required them to turn the feature on at the command-line
before they can use it.

Frankly, your use case is that you would prefer NUnit to work alike
with some competing frameworks in order to make it easier to create
training. That's not a terribly convincing case. NUnit is different
software. It has different features and that's part of what makes this
an interesting space.

OTOH, the use case of someone trying to test custom assertions is a
good one. It's precisely the reason we have the isolated context, in
fact. The logic is: I need a separate context to run this test because
the test result is in the context. I want tp examine the result in
that isolated context, not the result of the test itself - the one I'm
running. Unfortunately, I just don't have a good syntax right now to
expose it to users.

>>
>>
>> Hope this is helpful!
>
>
> It has been very helpful in confirming I wasn't going crazy! The last time I
> was this confused by any member of the xUnit family was when I tried to
> implement Lazy Setup in NUnit 2.0 and it didn't work. I spent hours in the
> debugger before convincing myself that I was doing things right and it was
> the framework that was leaking information between test methods. (I can't
> remember; has this been fixed in NUnit 3? Last I heard you were concern
> about breaking legacy tests that depended on it. ;-) )

Sorry, I don't know (or remember) what that's about. Did you file an
issue on it? Of course, we don't work on the V2 code base any longer,
but I believe we migrated all the issues that would still apply.

In any case, I don't undersrtand why you would want to implement Lazy
Setup for NUnit at all, since we have OneTimeSetUp for (shared)
fixtures.

> So what can I do right now to let me test my Custom Assertions. I teach this
> stuff to people in every course I run and I need to be able to tell them how
> to do it in NUnit 3.x; it wouldn't look very good to have NUnit 3 be the
> only member of the family in which we cannot test our Custom Assertions. (I
> already have to explain some of the other "foibles" of NUnit; I'd like the
> list of exceptions to get smaller, not larger!) I'm embarrassed to say that
> I'm still teaching based on NUnit 2.x (and running my exercises in 2.2.8)
> because of the inability to (meta)test my tests due to the changes in the
> framework after 2.2.8.

Well, as I'm sure you know, NUNit 2.2.8 was released in 2007. It's now
2017. That's a lot of change to swallow all at once!

It's clearly not impossible to (meta)test NUnit assertions after 2007,
since we've continued to do it all along. Feel free to ask for help
but asking us to not progress doesn't cut it.

My own approach would be to establish an isolated context (using the
ugly syntax or some new cleaner one we may subsequently invent) and
return the result of the assertion from it. Examine the fields of the
result to see if they are as you would expect. IMO, this is much
cleaner than catching an exception, especially since the exception has
much less information in it than the result. Of course, to do this,
you are looking at NUnit internals, but the exception is an internal
detail as well.

If you can get past the simple fact that NUnit does not include in
it's design goals "Use an implementation like what has been used since
vbUnit" then I'm happy to help you figure out how to test your stuff.

Charlie

[Charlie Poole]

Hi Gerard,

We got into such a big discussion around interface versus implementation
that I forgot to ask the most obvious question. If all you care about is the
exception, rather than the result, why not just use Assert.Throws?

[Test]
public void testAssertEquals_FlightDtosNotEqual()
{
FlightDto flightDto = CreateAnonymousFlightDto();
FlightDto differentFlightDto = CreateAnonymousFlightDto();

Assert.Throws<AssertionException>(() =>
AreEqual(flightDto, differentFlightDto, "2 FlightDto
should not be equal "));
}

This does exactly what you intended the example code to do. It works because
the Throws method understands the need to protect you from unintended results
caused by the exception.

Charlie

[Gerard Meszaros]

So I have done some more digging. I read through the Breaking Changes page again.

Definitely no mention of this breaking behaviour change. Then I read up the page on Assertions. 

Again, no mention of this change. In the Outline on the right, I saw a sub-page on Multiple Assertions. So I read that.

https://github.com/nunit/docs/wiki/Multiple-Asserts

Again, no mention of the breaking behaviour change.

 But I did find a 

[Gerard Meszaros]

Agggh. Google Groups interpretted a keystroke combination as Send! Sorry. and in trying to find my post, Groups finally decided to show me your 2 replies. So ignore this partial reply.:

Gerard

[Charlie Poole]

Hi Gerard,

Replying (anyway) to part of your partial reply...

As I was trying to explain... it's not in the docs that we no longer
throw an exception because I don't think it's in the docs that we ever
threw an exception. However, I could be wrong... as the docs are large
and many people get to edit them.

I'll wait for you to digest my replies.

Charlie

[Gerard Meszaros]

Thanks for pointing this out. I tried it and it does indeed work. It isn't obvious why this should work when try/catch doesn't  Does Assert.Throws create a new TestContext? That seems like a lot of overhead "just in case" the exception is an NUnit-specific one.

Before you sent this reply, I did find another workaround: Assert.Success seems to cancel the previous failure. The only weird side-effect of doing this is that the result for the test in the GUI in VS/Re# is "Success: <the message from the call>". But if I omit the message then it looks the same in the runner. 

Technically, I don't just care about the exception; I care that the test would be failed. Traditionally, throwing the exception was how the Assert method communicated that to the runner. Most Custom Asserts are the result of Extract Method on a bunch of duplicated asserts so they just aggregate a bunch of built-in Assert methods and those will still communicate correctly with the Runner.

One issue I ran into with converting to this idiom is that sometimes I want to assert on the message in the AssertError exception. Actually, I want to verify that the right message will be shown to the user but historically, it showed the message in the exception so it was sufficient to check the message. if the message will now be "stashed" by the Assert Method and not passed back through the exception than I will need to change this. If it is in the Exception, then how would I access it via Assert.Throws or Assert.That(lamda, Throws....)?

Gerard

[Gerard Meszaros]

I'm stripping out stuff that isn't relevant to make this more manageable in size.

On Saturday, February 4, 2017 at 1:40:13 AM UTC-7, charlie wrote:

Hi Gerard,

Replies inline...

On Fri, Feb 3, 2017 at 10:25 PM, Gerard Meszaros <gmka...@gmail.com> wrote:

I"m sure you aren't trying to say that NUnit is required to work
internally the same as VbUnit, et al.

No, I was merely pointing out that the only member of the xUnit family that I am aware of that exhibited the same behaviour as this was VbUnit back in the 1990's. So in that sense, NUnit 3 is regressing 20 years!

 

I realize that the difference
could be confusing for someone trying todo similar things in multiple
frameworks, but I don't think that's the case for most users.

It seems to me that most "full stack developers" need to program in more than one language so they need to use more than one unit test framework. Now, I don't build web apps in .Net so I don't know if this applies to typical .Net developers. But there are many developers who do work in both .Net and some other ecosystem. (e.g. Java, or PHP, or NodeJS ...)

 

Reporting the result of a failure isn't a side effect. It is the
effect of any assertion. In fact, I could just as well call the
throwing of an exception the side effect. The object of running a test
is to return a  result after all, not to throw an exception.

The benefits of side-effect free programming are fairly well understood. It is even enforced in some languages (e.g. Functional Programming). Side effects make programs harder to reason about.

 

>> The immediate cause of this change is the introduction of warning results
>> and multiple assertion failures per test.
>
>
> When was this breaking change introduced? I trolled through the list of
> breaking changes and didn't see it.

It was in 3.6, released a few weeks ago. I didn't put it in the
breaking changes page because I actually wasn't thinking of it as a
breaking change but as a change to an implementation detail. However,
since it can break some existing tests, I should add it there.

Agreed.

 

Well, you're a user and I just told you how to use it, so... yes!

The Internal namespace exists to tell people that they are using part
of the NUnit implementation, without actually declaring the class as
internal and making it inaccessible to them. People writing
extensions, for example, need to use Types in that namespace.

 

I'm assuming the implication of using the Internal namespace is that anything I use there could change in the next release.So I'm setting myself up for increase maintenance costs if I use it. Correct? 

Are you saying that NUnit is not going to throw AssertionError? How will you interrupt the execution of the test?

Unfortunately, I'm being telling people for years that if you expect a piece of code to throw an exception in response to some inputs you provide it, then you should write tests to check it. (And not to check for exceptions otherwise.) Exceptions are part of the interface exposed by the code.

 

OTOH, the use case of someone trying to test custom assertions is a
good one. It's precisely the reason we have the isolated context, in
fact. The logic is: I need a separate context to run this test because
the test result is in the context. I want tp examine the result in
that isolated context, not the result of the test itself - the one I'm
running. Unfortunately, I just don't have a good syntax right now to
expose it to users.

>>
>>
>> Hope this is helpful!
>
>
> It has been very helpful in confirming I wasn't going crazy! The last time I
> was this confused by any member of the xUnit family was when I tried to
> implement Lazy Setup in NUnit 2.0 and it didn't work. I spent hours in the
> debugger before convincing myself that I was doing things right and it was
> the framework that was leaking information between test methods.  (I can't
> remember; has this been fixed in NUnit 3? Last I heard you were concern
> about breaking legacy tests that depended on it. ;-)   )

Sorry, I don't know (or remember) what that's about. Did you file an
issue on it? Of course, we don't work on the V2 code base any longer,
but I believe we migrated all the issues that would still apply.  

This is known sometimes known as the "JUnit Instance Behaviour incompatability" 

 

In any case, I don't undersrtand why you would want to implement Lazy
Setup for NUnit at all, since we have OneTimeSetUp for (shared)
fixtures.

To understand why one should use a particular pattern, one needs to understand the alternative pattern. Until the various bookends were available, Lazy Fixture Setup was the only way to avoid the overhead of setting up fixtures. Now that most members of the xUnit family support some sort of bookend shared fixture setup/teardown methods, Lazy Initialization isn't used as much. But it is still useful to be aware of it. So I do mention it as a possibility. 

 

Well, as I'm sure you know, NUNit 2.2.8 was released in 2007. It's now
2017. That's a lot of change to swallow all at once!

 

And it's only going to get larger. Which is why I'm trying to catch up! But the effort involved in building a new MetaTest framework has kept me from doing it thus far.

If you can get past the simple fact that NUnit does not include in
it's design goals "Use an implementation like what has been used since
vbUnit" then I'm happy to help you figure out how to test your stuff.

I appreciate the offer. Unfortunately, I don't have the time to dig into this myself but I would be happy to contract this out to someone. I have posted a project on Freelancer.com to build me a MetaTest framework in NUnit 3.x. Not the Custom Assertions but the ability to run whole test suites inside another test and check the results using Assertions. And i would be happy to contribute it back to the NUnit3 project. If you know someone familiar with the internals of NUnit3 who might be interested in earning a few bucks doing this, feel free to put them in touch with me. Or if you (the project) want to build this capability into NUnit yourselves, I would be happy to make a modest financial contribution to the project. Say, $500 USD?

Thanks,

Gerard

[Charlie Poole]

Hi Gerard,

On Sat, Feb 4, 2017 at 10:39 AM, Gerard Meszaros <gmka...@gmail.com> wrote:
> Thanks for pointing this out. I tried it and it does indeed work. It isn't
> obvious why this should work when try/catch doesn't Does Assert.Throws
> create a new TestContext? That seems like a lot of overhead "just in case"
> the exception is an NUnit-specific one.

The overhead of creating a new context is very small. However, we really only
simulate the creation of one by plugging in a new result. And Assert.Throws
is not used in the average test suite very frequently.

>
> Before you sent this reply, I did find another workaround: Assert.Success
> seems to cancel the previous failure. The only weird side-effect of doing
> this is that the result for the test in the GUI in VS/Re# is "Success: <the
> message from the call>". But if I omit the message then it looks the same in
> the runner.

That's probably a bug to think about.

> Technically, I don't just care about the exception; I care that the test
> would be failed.

That was my original point.

> Traditionally, throwing the exception was how the Assert
> method communicated that to the runner.

True. Which makes this change a bit inconvenient. We had to change
how we test our own asserts.

> Most Custom Asserts are the result
> of Extract Method on a bunch of duplicated asserts so they just aggregate a
> bunch of built-in Assert methods and those will still communicate correctly
> with the Runner.

OK... small miscommunication there, as that's not what I mean by
"Custom Assert."
I'm on board with you now, however.

> One issue I ran into with converting to this idiom is that sometimes I want
> to assert on the message in the AssertError exception. Actually, I want to
> verify that the right message will be shown to the user but historically, it
> showed the message in the exception so it was sufficient to check the
> message. if the message will now be "stashed" by the Assert Method and not
> passed back through the exception than I will need to change this. If it is
> in the Exception, then how would I access it via Assert.Throws or
> Assert.That(lamda, Throws....)?

Assert.Throws returns the exception, if successful. So...

var ex = Assert.Throws...
Assert.That(ex.Message, Is.EqualTo("mymessage"));

The ThrowsConstraint can be chained with other constraints. So..

Assert.That(() => ...,
Throws<AssertionException>().With.Message.EqualTo("mymessage"));

For a single check, I prefer the second approach. For many checks, the first.

Charlie

[Charlie Poole]

Hi Gerard,

On Sat, Feb 4, 2017 at 11:05 AM, Gerard Meszaros <gmka...@gmail.com> wrote:
> I'm stripping out stuff that isn't relevant to make this more manageable in
> size.

I'll stay inline but strip out some more stuff...

>> I"m sure you aren't trying to say that NUnit is required to work
>> internally the same as VbUnit, et al.
>
>
> No, I was merely pointing out that the only member of the xUnit family that
> I am aware of that exhibited the same behaviour as this was VbUnit back in
> the 1990's. So in that sense, NUnit 3 is regressing 20 years!

That's a point of view. I've always considered use of exceptions for reporting
as a BadThing in unit test frameworks. Unfortunately, we need to use them
to terminate execution of the test arbitrarily. But we don't have to use them
at every point where the test has something to report, but is not necessarily
terminating.

I have no idea (thank God) how vbUnit worked. Our goal is not to go back
20 years, but to add new features that require some implementation changes.

> It seems to me that most "full stack developers" need to program in more
> than one language so they need to use more than one unit test framework.
> Now, I don't build web apps in .Net so I don't know if this applies to
> typical .Net developers. But there are many developers who do work in both
> .Net and some other ecosystem. (e.g. Java, or PHP, or NodeJS ...)

Most developers with a breadth of experience, in my experience, are able
to adapt to changes in how the tools work. Beginners coming from some
other language environment may have trouble, of course. We should try
to help them by explaining how things work.

> The benefits of side-effect free programming are fairly well understood. It
> is even enforced in some languages (e.g. Functional Programming). Side
> effects make programs harder to reason about.

That's true, but not always possible. Recording warnings and failures for
later reporting is a side effect. So is logging. Handling test events is a
side effect. NUnit is pretty much all side effects, since most test methods
return void.

For most users, it is not necessary to know what an AssertionException
even is in order to use NUnit.

>> Well, you're a user and I just told you how to use it, so... yes!
>>
>> The Internal namespace exists to tell people that they are using part
>> of the NUnit implementation, without actually declaring the class as
>> internal and making it inaccessible to them. People writing
>> extensions, for example, need to use Types in that namespace.
>
>
> I'm assuming the implication of using the Internal namespace is that
> anything I use there could change in the next release.So I'm setting myself
> up for increase maintenance costs if I use it. Correct?

This comes down to our misunderstanding about what "Custom Assert"
means. I use the term to mean something that makes use of NUnit's
extensibility to create a custom constraint and syntax that integrates
with NUnit's that invokes that constraint. You can't do that without
getting into the internal namespace.

You could see changes in that area, but we would try to minimize
any that impact (what we call) Custom Asserts. There's a back-burner
project to move some of those items that we support for extensibility
to a different namespace, so it's clearer.

Fun fact: I considered moving AssertionException to the Internal
namespace, where it really belongs. However, I knew that some
third-party runners made use of it so I didn't.

> Are you saying that NUnit is not going to throw AssertionError? How will you
> interrupt the execution of the test?

If the test needs interrupting, we will have to throw an exception.
I'm just saying
that many things happen that require recording do not require termination of
the test. Warnings are a current example. Recording _successful_ asserts
is a future example.

> Unfortunately, I'm being telling people for years that if you expect a piece
> of code to throw an exception in response to some inputs you provide it,
> then you should write tests to check it. (And not to check for exceptions
> otherwise.) Exceptions are part of the interface exposed by the code.

Absolutely. But how would that change if you took my point of view that
the exception is merely an implementation detail? What would the user
expect then? I think it would be something like "the test should fail with
a particular message."

>> Sorry, I don't know (or remember) what that's about. Did you file an
>> issue on it? Of course, we don't work on the V2 code base any longer,
>> but I believe we migrated all the issues that would still apply.
>
>
> This is known sometimes known as the "JUnit Instance Behaviour
> incompatability"

I understand that. It's a fact of NUnit life since Philip invented it in 2000.
It doesn't actually cause any problems, except as folks are transitioning
from Java. They catch on and the problem goes away.

I've suggested adding an optional approach where we use an instance
per test case. It didn't get a great deal of support. I think the idea doesn't
particularly have legs because the problems associated with it only arise
if you program NUnit as if it were JUnit, which most users don't do.

>> In any case, I don't undersrtand why you would want to implement Lazy
>> Setup for NUnit at all, since we have OneTimeSetUp for (shared)
>> fixtures.
>
>
> To understand why one should use a particular pattern, one needs to
> understand the alternative pattern.

Indeed!

> Until the various bookends were
> available, Lazy Fixture Setup was the only way to avoid the overhead of
> setting up fixtures. Now that most members of the xUnit family support some
> sort of bookend shared fixture setup/teardown methods, Lazy Initialization
> isn't used as much. But it is still useful to be aware of it. So I do
> mention it as a possibility.

Right. OneTimeSetUp _is_ Lazy SetUp.

>> Well, as I'm sure you know, NUNit 2.2.8 was released in 2007. It's now
>> 2017. That's a lot of change to swallow all at once!
>
>
> And it's only going to get larger. Which is why I'm trying to catch up! But
> the effort involved in building a new MetaTest framework has kept me from
> doing it thus far.

The problem you ran into is quite obscure - because it is a meta-test.
If you have some other way to get familiar with NUnit 3.6 doing regular
non-meta testing, I think it might go easier.

Regarding meta-testing... couldn't you have some api like

ExpectFailure(() => /*code that should cause a failure */);

and keep the implementation (and details of syntax) specific to each framework?

>>
>> If you can get past the simple fact that NUnit does not include in
>> it's design goals "Use an implementation like what has been used since
>> vbUnit" then I'm happy to help you figure out how to test your stuff.
>
>
> I appreciate the offer. Unfortunately, I don't have the time to dig into
> this myself but I would be happy to contract this out to someone. I have
> posted a project on Freelancer.com to build me a MetaTest framework in NUnit
> 3.x. Not the Custom Assertions but the ability to run whole test suites
> inside another test and check the results using Assertions. And i would be
> happy to contribute it back to the NUnit3 project. If you know someone
> familiar with the internals of NUnit3 who might be interested in earning a
> few bucks doing this, feel free to put them in touch with me. Or if you (the
> project) want to build this capability into NUnit yourselves, I would be
> happy to make a modest financial contribution to the project. Say, $500 USD?

Is there an API? Something I can see without joining freelancer.com?

Charlie

[Gerard Meszaros]

The API and acceptance test is the MetaTest:

        [Test]

        public void TestExercise07Result()

        {

            TestResult result = GetTestResultsForDLL("Exercise-07-Result.dll");

            verifyResultCounts(result, 8, 2, 0);

            assertTestCaseFailedWithMessageParts(   "TestClassName", "TestMethodName", 

                                                    "expected: <\"<span class=\"tinyBoldText\">12:01 AM</span>\">",

                                                    "but was: <\"<span class=\"tinyBoldText\">Midnight</span>\">"  );

            assertTestCaseFailedEquality( "TestClassName",  "TestMethodName",  "expected",  "actual");

        }

where:

       protected void assertTestCaseFailedEquality(string TestClassName, string TestMethodName, string expected, string actual)

       protected void assertTestCaseFailedWithMessageParts(string TestClassName, string TestMethodName, string MessagePart1, string MessagePart2, string MessagePart3)

The exact syntax is not important. What is important is the ability to:

1. Execute an entire test project with a single statement and get back some kind of result object.
2. Methods on the overall result object to return the summary statistics (Run, Passed, Failed, etc.) so these can be compared to expected values.
3. Methods on the overall result object to retrieve specific test failures by TestClass name and TestMethod name
4. Methods on the specific test failures to interogate the specifics of the failure so they can be compared with expected values.

And all this needs to be done within [Test] methods to make them easy to run and report results. If there is an existing internal API that could be exposed (e.g. NUnit.Framework.MetaTest) then I could build my DSL (as in the example above) on top of it. Ideally, it wouldbn't be "Internal" because I don't want to have to revisit it each a new version of NUnit is created.

I would imagine that the code required to do this all exists internally because TestRunners need to do more-or-less the same thing. The main difference is that this would all be running inside another NUnit TestRunner. And that I would be asserting on the results rather than just rendering them to some kind of view. Here's what they look like. Each [Test] runs an entire test VSProj:

The code sample above corresponds to 1 line in the output.

Is this clear enough? Feel free to ask more questions.

Thanks,

Gerard

[Gerard Meszaros]

Charlie,

Just wondering if you had a chance to look at this.

Thanks,

Gerard

[Gerard Meszaros]

Just wondering if you wanted the donation to the project or should I hire someone to build the meta-test runner for me?

Thanks,

Gerard

On Saturday, February 4, 2017 at 1:28:58 PM UTC-7, charlie wrote:

[Charlie Poole]

Hi Gerard,

Nobody has come forward and although I'm interested in the outcome I
don't have the time to do it. Folks? Last chance to respond to this
thread!

Regarding the original discussion that got us started here, you should
take a look at the action proposed to resolve it at
https://github.com/nunit/nunit/issues/2043

Charlie

[Gerard Meszaros]

Makes Sense to me. Thanks for pointing this out to me.

Gerard

[Gerard Meszaros]

Just to clarify to anyone reading this:

1. I want to be able to run a test runner that returns the results of the tests in an easily digested form. In NUnit 2.2.8, the easiest way to do this was by running the test project DLL but I'd be find with running a test class or a namespace or whatever is easiest.
2. I want to be be able to do this inside an NUnit [Test] method. 
3. I need to be able to assert the that right number of tests ran, failed (and error'ed if that distinction survives).
4. I need to be able to assert on the message text that a user would see. Ideally, the raw message and not subject to formatting (which might change from release to release)
5. ideally, I'd like to be able to run tests built in previous versions on NUnit. But not a deal-breaker if it turns out to be hard to do.

I suspect running the set of tests is the easy part; after all all the various TestRunners do this. Probably the more interesting part is how to interrogate the results.  

I'm offering a bounty of $500 USD for this. Either for the person doing it or as a donation to the NUnit project. Your choice.

If no-one here wants to do the work, I'll try to hire someone to do it on Freelancer.com. Any suggestions of approaches or people would be appreciated.

Thanks,

Gerard

On Thursday, February 16, 2017 at 5:18:51 PM UTC-7, charlie wrote:

Hi Gerard,

Nobody has come forward and although I'm interested in the outcome I
don't have the time to do it. Folks? Last chance to respond to this
thread!

[Charlie Poole]

Hi Gerard,

Some comments for you and/or whomever picks this up.

On Thu, Feb 16, 2017 at 9:27 PM, Gerard Meszaros <gmka...@gmail.com> wrote:
> Just to clarify to anyone reading this:
>
> I want to be able to run a test runner that returns the results of the tests
> in an easily digested form. In NUnit 2.2.8, the easiest way to do this
was
> by running the test project DLL but I'd be find with running a test class or
> a namespace or whatever is easiest.

The NUnit tests contain utility methods that run either a test class or a method
in the TestUtilities namespace. There is code in several tests that runs and
verifies the result of the tests in our mock-assembly... search
"mock-assembly.dll."

> I want to be be able to do this inside an NUnit [Test] method.
> I need to be able to assert the that right number of tests ran, failed (and
> error'ed if that distinction survives).

That info is available at the top level of the result tree that is
returned, just as
in older versions of NUnit.

> I need to be able to assert on the message text that a user would see.
> Ideally, the raw message and not subject to formatting (which might change
> from release to release)

There are methods in TestUtilities to navigate to a particular named test. Once

there, you have access to the raw message that is provided to the runner.

> ideally, I'd like to be able to run tests built in previous versions on
> NUnit. But not a deal-breaker if it turns out to be hard to do.

Because your own tests would be loading a version of the framework, the user
tests would have to be using the same one for all the techniques discussed
above. If you really wanted independence, I think you would have to launch
the console runner and load the xml result file into memory. That would mean
dealing with the XML rather than with an object representation of the result.

Charlie