Re: [nunit-discuss] NUnit 2.6, TestCaseSource and exception handling for tests
739 views
Skip to first unread message
Charlie Poole
Jan 11, 2013, 4:17:31 PM1/11/13
to NUnit-Discuss
As mentioned in the other thread, adding new parameters to
ExpectedException or TestCaseData is something we want to get away
from. Some people would like to have an expected message, some people
would like to access the parameter name of an ArgumentNull exception,
some people would like the hresult. It's an endless task.
In the case of ExpectedException, we went a long way toward trying to
make folks happy, but eventually realized that we were just making the
API more and more complex, with insufficient benefit.
In fact, testing the result of an exception with Assert.Throws is much
superior to using ExpectedException attribute or the equivalent
TestCaseData syntax. When you make the test inline, you know exactly
where in the code your exception was thrown whereas use of
ExpectedException only tells you it was thrown somewhere in the method
- and maybe not from the place where you imagine it was thrown.
However, with all that said, I'd like to give some consideration to
what you want. So... what do you want? That is, what syntax would you
like to use, if it existed.
Charlie
On Fri, Jan 11, 2013 at 12:58 PM, Stecy Dube <stecy...@gmail.com> wrote:
> Hello to all,
>
> Not sure if it is something I missed but here's what I have:
>
> [TestFixture]
> public class MyFixtureTests
> {
> private class MySource : IEnumerable
> {
> private readonly TestCaseData[] _testCases = new[]
> {
> new TestCaseData (20, 2).Returns (10),
> new TestCaseData (1, 0).Throws (typeof
> (DivideByZeroException))
> };
>
> public IEnumerator GetEnumerator ()
> {
> return _testCases.GetEnumerator ();
> }
> }
>
> [TestCaseSource (typeof (MySource))]
> public int TestDivision (int a, int b)
> {
> return a/b;
> }
> }
>
>
> However, I cannot specify additional conditions for the exception test like
> I could with Assert.Throws()
>
> var ex = Assert.Throws<ArgumentNullException> (() => ......);
> Assert.That (ex.ParamName, Is.EqualTo ("param1"));
>
>
> Could it be considered to add this feature to NUnit 2.6 ?
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/UQ-nGbEmUo8J.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to
> nunit-discus...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nunit-discuss?hl=en.
ExpectedException or TestCaseData is something we want to get away
from. Some people would like to have an expected message, some people
would like to access the parameter name of an ArgumentNull exception,
some people would like the hresult. It's an endless task.
In the case of ExpectedException, we went a long way toward trying to
make folks happy, but eventually realized that we were just making the
API more and more complex, with insufficient benefit.
In fact, testing the result of an exception with Assert.Throws is much
superior to using ExpectedException attribute or the equivalent
TestCaseData syntax. When you make the test inline, you know exactly
where in the code your exception was thrown whereas use of
ExpectedException only tells you it was thrown somewhere in the method
- and maybe not from the place where you imagine it was thrown.
However, with all that said, I'd like to give some consideration to
what you want. So... what do you want? That is, what syntax would you
like to use, if it existed.
Charlie
On Fri, Jan 11, 2013 at 12:58 PM, Stecy Dube <stecy...@gmail.com> wrote:
> Hello to all,
>
> Not sure if it is something I missed but here's what I have:
>
> [TestFixture]
> public class MyFixtureTests
> {
> private class MySource : IEnumerable
> {
> private readonly TestCaseData[] _testCases = new[]
> {
> new TestCaseData (20, 2).Returns (10),
> new TestCaseData (1, 0).Throws (typeof
> (DivideByZeroException))
> };
>
> public IEnumerator GetEnumerator ()
> {
> return _testCases.GetEnumerator ();
> }
> }
>
> [TestCaseSource (typeof (MySource))]
> public int TestDivision (int a, int b)
> {
> return a/b;
> }
> }
>
>
> However, I cannot specify additional conditions for the exception test like
> I could with Assert.Throws()
>
> var ex = Assert.Throws<ArgumentNullException> (() => ......);
> Assert.That (ex.ParamName, Is.EqualTo ("param1"));
>
>
> Could it be considered to add this feature to NUnit 2.6 ?
>
> Thank you.
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/UQ-nGbEmUo8J.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to
> nunit-discus...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/nunit-discuss?hl=en.
Stecy Dube
Jan 11, 2013, 4:46:39 PM1/11/13
to nunit-...@googlegroups.com
I hadn't realized that adding more stuff to TestCaseData would be the same as the ExpectedExceptionAttribute.
I didn't like that attribute back then and it was great you added Assert.Throws<> instead, for all the reasons you mentioned.
So, let's forget about polluting TestCaseData with more "variations".
However, I like the fact that I can have one test method having multiple input parameters and using TestCaseSource. Saves on the number of methods to manage and also eliminates redundancy. Gotta love DRY... ;)
Now that you've opened my eyes, I don't know what could be the best way or even if there's a better way.
Brainstorming about it, how about passing a lambda for the exception check? That way, someone could code whatever they want for the conditions?
Something like:
.Throws<ArgumentNullException> (ex => ex.ParamName == "p1");
But it still doesn't protect us if the exception is generated from an unexpected place in the test code. oh well...
As an aside, the Assert.That (testDelegate, Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as TestCaseData and could also be "cured" with the lambda trick.
I didn't like that attribute back then and it was great you added Assert.Throws<> instead, for all the reasons you mentioned.
So, let's forget about polluting TestCaseData with more "variations".
However, I like the fact that I can have one test method having multiple input parameters and using TestCaseSource. Saves on the number of methods to manage and also eliminates redundancy. Gotta love DRY... ;)
Now that you've opened my eyes, I don't know what could be the best way or even if there's a better way.
Brainstorming about it, how about passing a lambda for the exception check? That way, someone could code whatever they want for the conditions?
Something like:
.Throws<ArgumentNullException> (ex => ex.ParamName == "p1");
But it still doesn't protect us if the exception is generated from an unexpected place in the test code. oh well...
As an aside, the Assert.That (testDelegate, Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as TestCaseData and could also be "cured" with the lambda trick.
Charlie Poole
Jan 11, 2013, 6:06:58 PM1/11/13
to NUnit-Discuss
Hi Stecy,
On Fri, Jan 11, 2013 at 1:46 PM, Stecy Dube <stecy...@gmail.com> wrote:
> I hadn't realized that adding more stuff to TestCaseData would be the same
> as the ExpectedExceptionAttribute.
> I didn't like that attribute back then and it was great you added
> Assert.Throws<> instead, for all the reasons you mentioned.
>
> So, let's forget about polluting TestCaseData with more "variations".
In the end, I think ExpectedExceptionAttribute, TestCaseAttribute and
TestCaseData
should all support the same properties. It's possible that the list of
what's supported
could be restricted in NUnit 3.0, since it's a major change release.
Of course some things (like lambdas) can't be supported on attributes, so that
would be an exception.
> However, I like the fact that I can have one test method having multiple
> input parameters and using TestCaseSource. Saves on the number of methods to
> manage and also eliminates redundancy. Gotta love DRY... ;)
>
> Now that you've opened my eyes, I don't know what could be the best way or
> even if there's a better way.
>
> Brainstorming about it, how about passing a lambda for the exception check?
> That way, someone could code whatever they want for the conditions?
> Something like:
> .Throws<ArgumentNullException> (ex => ex.ParamName == "p1");
I've been thinkng about t hat as a possibility myself. It could be added to
both Assert.Throws and TestCaseSource.
> But it still doesn't protect us if the exception is generated from an
> unexpected place in the test code. oh well...
I think that anticipating an exception at the level of the test, rather than
at a specific line is something that deserves a warning like "Use this
carefully" but providing the facility isn't evil in itself. After all, some
test methods may only have a single line!
> As an aside, the Assert.That (testDelegate,
> Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as
> TestCaseData and could also be "cured" with the lambda trick.
Except you _can_ write
Assert.That( testDelegate,
Throws.Typeof<ArgumentNullException>.With.Message.EqualTo("some
string");
Charlie
On Fri, Jan 11, 2013 at 1:46 PM, Stecy Dube <stecy...@gmail.com> wrote:
> I hadn't realized that adding more stuff to TestCaseData would be the same
> as the ExpectedExceptionAttribute.
> I didn't like that attribute back then and it was great you added
> Assert.Throws<> instead, for all the reasons you mentioned.
>
> So, let's forget about polluting TestCaseData with more "variations".
TestCaseData
should all support the same properties. It's possible that the list of
what's supported
could be restricted in NUnit 3.0, since it's a major change release.
Of course some things (like lambdas) can't be supported on attributes, so that
would be an exception.
> However, I like the fact that I can have one test method having multiple
> input parameters and using TestCaseSource. Saves on the number of methods to
> manage and also eliminates redundancy. Gotta love DRY... ;)
>
> Now that you've opened my eyes, I don't know what could be the best way or
> even if there's a better way.
>
> Brainstorming about it, how about passing a lambda for the exception check?
> That way, someone could code whatever they want for the conditions?
> Something like:
> .Throws<ArgumentNullException> (ex => ex.ParamName == "p1");
both Assert.Throws and TestCaseSource.
> But it still doesn't protect us if the exception is generated from an
> unexpected place in the test code. oh well...
at a specific line is something that deserves a warning like "Use this
carefully" but providing the facility isn't evil in itself. After all, some
test methods may only have a single line!
> As an aside, the Assert.That (testDelegate,
> Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as
> TestCaseData and could also be "cured" with the lambda trick.
Assert.That( testDelegate,
Throws.Typeof<ArgumentNullException>.With.Message.EqualTo("some
string");
Charlie
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/lpCkZQNfbCkJ.
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
Stecy Dube
Jan 13, 2013, 11:47:47 AM1/13/13
to nunit-...@googlegroups.com
In the end, I think ExpectedExceptionAttribute, TestCaseAttribute and
TestCaseData
should all support the same properties. It's possible that the list of
what's supported
could be restricted in NUnit 3.0, since it's a major change release.
I thought that the ExpectedExceptionAttribute would be deprecated in favor of the Assert.Throws syntax in NUnit 3 since it is not really useful to set the expectation in an attribute IMHO.
> As an aside, the Assert.That (testDelegate,
> Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as
> TestCaseData and could also be "cured" with the lambda trick.
Except you _can_ write
Assert.That( testDelegate,
Throws.Typeof<ArgumentNullException>.With.Message.EqualTo("some
string");
Assert.That (testDelegate, Throws.Typeof<ArgumentNullException>.With.ParamName.EqualTo("aParam");
someone have to use With.Property("ParamName") instead and that is not refactory friendly.
And I know that it is not desirable to have all the possibilities upfront so it might be useful to have lambdas on the Property() method also.
Any chance of having lambdas in the next version of NUnit 2.6 and, of course, in v3?
Anyhow, thanks for the support.
Charlie Poole
Jan 13, 2013, 12:35:36 PM1/13/13
to NUnit-Discuss
Hi Stecy,
On Sun, Jan 13, 2013 at 8:47 AM, Stecy Dube <stecy...@gmail.com> wrote:
>
>> In the end, I think ExpectedExceptionAttribute, TestCaseAttribute and
>> TestCaseData
>> should all support the same properties. It's possible that the list of
>> what's supported
>> could be restricted in NUnit 3.0, since it's a major change release.
>>
>
> I thought that the ExpectedExceptionAttribute would be deprecated in favor
> of the Assert.Throws syntax in NUnit 3 since it is not really useful to set
> the expectation in an attribute IMHO.
If I used the word deprecated, I misspoke. When we deprecate something
in the API, we mark it as obsolete and later remove it. For example, we
deprecated use of Is.InstanceOfType, replacing it with Is.InstanceOf.
In the case of ExpectedExceptionAttribute, it's a technique I don't recommend
as a coach, except in limited circumstances. But deprecating and then removing
it would amount to enforcing my view on users, who have come to expect such
an attribute in a unit test framework. Anyway, it is useful in some
circumstances.
So ExpectedExceptionAttribute will remain in NUnit for the forseeable future but
will any enhancements will go into Assert.Throws instead. I do think we should
add features to TestCaseAttribute and TestCaseData to match what is provided
by ExpectedException - otherwise it's confusing. Users wonder "Why can I
specify the message in this case, but not that case?" for example.
>> > As an aside, the Assert.That (testDelegate,
>> > Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as
>> > TestCaseData and could also be "cured" with the lambda trick.
>>
>> Except you _can_ write
>> Assert.That( testDelegate,
>> Throws.Typeof<ArgumentNullException>.With.Message.EqualTo("some
>> string");
>
>
> But you can't write something like:
> Assert.That (testDelegate,
> Throws.Typeof<ArgumentNullException>.With.ParamName.EqualTo("aParam");
> someone have to use With.Property("ParamName") instead and that is not
> refactory friendly.
>
> And I know that it is not desirable to have all the possibilities upfront so
> it might be useful to have lambdas on the Property() method also.
Adding them to the Property() method just adds them in one place. It's
better to use a separate Syntactic element for lambdas. See next...
> Any chance of having lambdas in the next version of NUnit 2.6 and, of
> course, in v3?
Try writing this...
Assert.That(testDelegate, Throws<ArgumentNullException>.Matches(...));
using a lambda expression. I'm not in the code right now, so you may need
to add .And. in there to make it work. If it doesn't work, you can file a bug
on NUnit 2.6.
Charlie
> Anyhow, thanks for the support.
>
On Sun, Jan 13, 2013 at 8:47 AM, Stecy Dube <stecy...@gmail.com> wrote:
>
>> In the end, I think ExpectedExceptionAttribute, TestCaseAttribute and
>> TestCaseData
>> should all support the same properties. It's possible that the list of
>> what's supported
>> could be restricted in NUnit 3.0, since it's a major change release.
>>
>
> I thought that the ExpectedExceptionAttribute would be deprecated in favor
> of the Assert.Throws syntax in NUnit 3 since it is not really useful to set
> the expectation in an attribute IMHO.
in the API, we mark it as obsolete and later remove it. For example, we
deprecated use of Is.InstanceOfType, replacing it with Is.InstanceOf.
In the case of ExpectedExceptionAttribute, it's a technique I don't recommend
as a coach, except in limited circumstances. But deprecating and then removing
it would amount to enforcing my view on users, who have come to expect such
an attribute in a unit test framework. Anyway, it is useful in some
circumstances.
So ExpectedExceptionAttribute will remain in NUnit for the forseeable future but
will any enhancements will go into Assert.Throws instead. I do think we should
add features to TestCaseAttribute and TestCaseData to match what is provided
by ExpectedException - otherwise it's confusing. Users wonder "Why can I
specify the message in this case, but not that case?" for example.
>> > As an aside, the Assert.That (testDelegate,
>> > Throws.TypeOf<ArgumentNullException>()) suffers from the same problem as
>> > TestCaseData and could also be "cured" with the lambda trick.
>>
>> Except you _can_ write
>> Assert.That( testDelegate,
>> Throws.Typeof<ArgumentNullException>.With.Message.EqualTo("some
>> string");
>
>
> But you can't write something like:
> Assert.That (testDelegate,
> Throws.Typeof<ArgumentNullException>.With.ParamName.EqualTo("aParam");
> someone have to use With.Property("ParamName") instead and that is not
> refactory friendly.
>
> And I know that it is not desirable to have all the possibilities upfront so
> it might be useful to have lambdas on the Property() method also.
better to use a separate Syntactic element for lambdas. See next...
> Any chance of having lambdas in the next version of NUnit 2.6 and, of
> course, in v3?
Assert.That(testDelegate, Throws<ArgumentNullException>.Matches(...));
using a lambda expression. I'm not in the code right now, so you may need
to add .And. in there to make it work. If it doesn't work, you can file a bug
on NUnit 2.6.
Charlie
> Anyhow, thanks for the support.
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/iIAjsMpJTikJ.
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
Stecy Dube
Jan 14, 2013, 11:09:31 AM1/14/13
to nunit-...@googlegroups.com
Try writing this...
Assert.That(testDelegate, Throws<ArgumentNullException>.Matches(...));
using a lambda expression. I'm not in the code right now, so you may need
to add .And. in there to make it work. If it doesn't work, you can file a bug
on NUnit 2.6.
I had to use
Assert.That (testDelegate, Throws.ArgumentException.And.Matches<ArgumentException> (x => x.ParamName == "name"));It works but the error text reported when the assert is not satisfied is rather generic:
Expected: <System.ArgumentException> and value matching lambda expression
But that will suffice for now as it is a cleaner workaround regarding refactoring.
However, there should be an easier way to express the intent and have a better error message, something like:
Assert.That (testDelegate, Throws.ArgumentException.And.Property(ex => ex.ParamName).EqualTo ("name"));
with the error reporting:
Expected: Property lambda ex.ParamName with value "name"
but got
Property lambda ex.ParamName with value "myParam"
I don't know if it can be done easily but what do you think about this solution?
Charlie Poole
Jan 14, 2013, 12:24:54 PM1/14/13
to NUnit-Discuss
Hi Stecy,
That's possible. If you file a bug / feature request on at
http://bugs.launchpad.net/nunitv2, I'll give it a shot.
Charlie
That's possible. If you file a bug / feature request on at
http://bugs.launchpad.net/nunitv2, I'll give it a shot.
Charlie
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/zXmYbNA6eWgJ.
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
Stecy Dube
Jan 14, 2013, 2:02:04 PM1/14/13
to nunit-...@googlegroups.com
Done !
Bug #1099517
Bug #1099517
Charlie Poole
Jan 14, 2013, 3:07:40 PM1/14/13
to NUnit-Discuss
Thanks... I'll experiment with the idea.
Charlie
On Mon, Jan 14, 2013 at 11:02 AM, Stecy Dube <stecy...@gmail.com> wrote:
> Done !
>
> Bug #1099517
Charlie
On Mon, Jan 14, 2013 at 11:02 AM, Stecy Dube <stecy...@gmail.com> wrote:
> Done !
>
> Bug #1099517
>
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/DSq6RdnlyDoJ.
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
Charlie Poole
Jan 14, 2013, 6:25:09 PM1/14/13
to NUnit-Discuss
Hi Anthony,
Actually, Stecy was starting from that statement, but trying to
eliminate the need to specify the property name as a string, which
hampers refactoring.
The alternative formulation (which I suggested) solves that but
introduces a new problem: a bad error message.
In NUnit 3.0, we plan to use Expressions to give a better error
message in a situation like the above.
Charlie
On Mon, Jan 14, 2013 at 2:43 PM, Anthony Langsworth
<anthony.l...@gmail.com> wrote:
> Can you work around the issue of the vague error message with (I do not have
> a compiler handy so there may be typos in the code below):
>
> Assert.That(testDelegate,
> Throws.ArgumentException.And.Property("ParamName").EqualTo("name"));
>
> Although you lose some of the flexibility with the predicate passed to
> Matches, if the property value differs you will get a more descriptive error
> message.
Actually, Stecy was starting from that statement, but trying to
eliminate the need to specify the property name as a string, which
hampers refactoring.
The alternative formulation (which I suggested) solves that but
introduces a new problem: a bad error message.
In NUnit 3.0, we plan to use Expressions to give a better error
message in a situation like the above.
Charlie
On Mon, Jan 14, 2013 at 2:43 PM, Anthony Langsworth
<anthony.l...@gmail.com> wrote:
> Can you work around the issue of the vague error message with (I do not have
> a compiler handy so there may be typos in the code below):
>
> Assert.That(testDelegate,
> Throws.ArgumentException.And.Property("ParamName").EqualTo("name"));
>
> Although you lose some of the flexibility with the predicate passed to
> Matches, if the property value differs you will get a more descriptive error
> message.
> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/KQ7DppKLhs0J.
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To view this discussion on the web visit
Anthony Langsworth
Jan 15, 2013, 5:40:56 AM1/15/13
to nunit-...@googlegroups.com
My apologies for not reading the thread all the way through. I look forward to the NUnit 3.0 implementation.
I suppose this is really a C# language deficiency. Moving away from automated unit testing for a moment, if I want to throw an ArgumentNullException the code should create it with the parameter name but, not withstanding the solutions provided in http://stackoverflow.com/questions/869610/resolving-a-parameter-name-at-runtime, there is no clean way to get it (like typeof for compile time type determination and .GetType() for runtime). This means an argument rename may lead to an incorrect ParamName property in the exception, just like in Stecy's case.
Reply all
Reply to author
Forward
0 new messages