Support 3rd party assertion libs

[Kenneth Xu]

Hello, this is a feature request.

As more and more test assertion libs rise from the horizon, we'll soon be able to choose our favorite assertion syntax to use with NUnit. Just like we are able to use our favorite mocking frameworks.

You can use 3rd party assertion libs with NUnit today. Assertion libs typically throw exception when assertion fails, and NUnit show it just like any other exception failure.

Below is an example using ExpressionToCode with NUnit. You can find that it comes with noises like the name of the exception class and the stack frames from ExpressionToCodeLib.

Test 'ExpressionAssert.WithExpressionToCode.Tests.TryPAssertTest.BooleanAssertWithMessage' failed: ExpressionToCodeLib.PAssertFailedException : My message

Enumerable.Range(0, 1000).ToDictionary(i => "n" + (object)i)["n3"].ToString() == 3.5.ToString()
             |                 |                            |         |        |        |
             |                 |                            |         |        |        "3.5"
             |                 |                            |         |        false
             |                 |                            |         "3"
             |                 |                            3
             |                 {[n0, 0], [n1, 1], [n2, 2], [n3, 3], [n4, 4], [n5, 5], [n6, 6], [n7, 7], [n8, 8], [n9, 9], ...}
             {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...}
    at ExpressionToCodeLib.PAssert.That(Expression`1 assertion, String msg)
    TryPAssertTest.cs(21,0): at ExpressionAssert.WithExpressionToCode.Tests.TryPAssertTest.BooleanAssertWithMessage()

Ideally, above example should be reported as assertion failure instead of exception failure. In other words, the PAssertionFailedException should be treated just like NUnit's own AssertionException.

Test 'ExpressionAssert.WithExpressionToCode.Tests.TryPAssertTest.BooleanAssertWithMessage' failed:
My message

Enumerable.Range(0, 1000).ToDictionary(i => "n" + (object)i)["n3"].ToString() == 3.5.ToString()
             |                 |                            |         |        |        |
             |                 |                            |         |        |        "3.5"
             |                 |                            |         |        false
             |                 |                            |         "3"
             |                 |                            3
             |                 {[n0, 0], [n1, 1], [n2, 2], [n3, 3], [n4, 4], [n5, 5], [n6, 6], [n7, 7], [n8, 8], [n9, 9], ...}
             {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, ...}
    TryPAssertTest.cs(21,0): at ExpressionAssert.WithExpressionToCode.Tests.TryPAssertTest.BooleanAssertWithMessage()

To achieve this, we can introduce an assembly level attribute, say AssertionExceptionAttribute, that takes Type as argument. e.g.

[assembly:NUnit.Framework.AssertionException(typeof(ExpressionToCodeLib.PAssertFailedException))]

NUnit can scan test assembly for the attribute and register the specified exception type. So when it encounters such exception during test, it can output message the same way as AssertionException does and remove the stack frames from the assembly where the registered exception resides.

Further, NUnit can dynamically pre-register some assertion exceptions from well known assertion libs. In such a way it can provide out of box support to those assertion libs without strongly referencing to them.

Thoughts?

Cheers,
Kenneth

[Charlie Poole]

Hi Kenneth,

This generally makes sense to me. It's similar to something I have
thought about doing
for the exceptions generated by well-known mocking frameworks. Further
comments inline...

On Mon, Oct 17, 2011 at 10:59 PM, Kenneth Xu <kenn...@gmail.com> wrote:
> Hello, this is a feature request.
>
> As more and more test assertion libs rise from the horizon, we'll soon be
> able to choose our favorite assertion syntax to use with NUnit. Just like we
> are able to use our favorite mocking frameworks.
>
> You can use 3rd party assertion libs with NUnit today. Assertion libs
> typically throw exception when assertion fails, and NUnit show it just like
> any other exception failure.

Note that any library that already uses NUnit could also throw an NUnit
AssertionException, rather than it's own exception.

I hadn't thought about the stack trace. That's certainly doable as
well, although
we might need more info about what to exclude.

> Further, NUnit can dynamically pre-register some assertion exceptions from
> well known assertion libs. In such a way it can provide out of box support
> to those assertion libs without strongly referencing to them.

Another possibility is for the third-party library to apply an
attribute known to
NUnit to it's exception classes.

> Thoughts?

This sounds reasonably doable for 2.6, so long as we avoid any need for NUnit
to have a reference to the exception class - we would need to use a
string rather
than a typeof expression with the attribute. For 3.0, we can use the
type itself.

Charlie
> Cheers,
> Kenneth
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to
> nunit-discus...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nunit-discuss?hl=en.
>

[Kenneth Xu]

Hi Charlie,

On Tue, Oct 18, 2011 at 7:44 PM, Charlie Poole <nuni...@gmail.com> wrote:

Hi Kenneth,

This generally makes sense to me. It's similar to something I have
thought about doing
for the exceptions generated by well-known mocking frameworks. Further
comments inline...

Exactly!
 

On Mon, Oct 17, 2011 at 10:59 PM, Kenneth Xu <kenn...@gmail.com> wrote:
> Hello, this is a feature request.
>
> As more and more test assertion libs rise from the horizon, we'll soon be
> able to choose our favorite assertion syntax to use with NUnit. Just like we
> are able to use our favorite mocking frameworks.
>
> You can use 3rd party assertion libs with NUnit today. Assertion libs
> typically throw exception when assertion fails, and NUnit show it just like
> any other exception failure.

Note that any library that already uses NUnit could also throw an NUnit
AssertionException, rather than it's own exception.

Yes! Satisfyr has a version specific for NUnit. But others typically avoided the hard dependency so the same lib can be used under various unit test frameworks.

Maybe an regex matching string to filter out frames at the top.
[assembly:NUnit.Framework.AssertionException(typeof(ExpressionToCodeLib.PAssertFailedException), StackReductionFilter="ExpressionToCodeLib")]

> Further, NUnit can dynamically pre-register some assertion exceptions from
> well known assertion libs. In such a way it can provide out of box support
> to those assertion libs without strongly referencing to them.

Another possibility is for the third-party library to apply an
attribute known to
NUnit to it's exception classes.

 

I'm not sure if I understand this. Do you mean an attribute defined in NUnit, an attribute in somewhere that NUnit reference to or any attribute follows a specific naming convention? In any case, it would require assertion lib auther to coordinate.

> Thoughts?

This sounds reasonably doable for 2.6, so long as we avoid any need for NUnit
to have a reference to the exception class - we would need to use a
string rather
than a typeof expression with the attribute. For 3.0, we can use the
type itself.

That's what I meant by "dynamically". There shouldn't any hard reference so typeof won't work. An alternative to pure string comparison is to load a Type object dynamically using assembly/class name. Then use Type.IsAssignableFrom. This helps when 3rd-party lib uses exception hierarchy.

Cheers,
Kenneth