Shouldly testing library

[Dave Newman]

Hey Guys,

I'm playing around with a new "should" style of testing with a focus
on better error messages. Or error messages with context!
It's based on NUnit and RhinoMocks, so for comparison this is the
current Assert way:

Assert.That(contestant.Points, Is.EqualTo(1337));

For your troubles, you get this message, when it fails:

Expected 1337 but was 0

With Shouldly:

contestant.Points.ShouldBe(1337);

Which is just syntax, so far, but the message when it fails:

contestant.Points should be 1337 but was 0

It might be easy to underestimate how useful this is. Another example,
side by side:

Assert.That(map.IndexOfValue("boo"), Is.EqualTo(2)); // ->
Expected 2 but was 1
map.IndexOfValue("boo").ShouldBe(2); // ->
map.IndexOfValue("boo") should be 2 but was 1

Shouldly uses the source code at the ShouldBe statement to report on
errors, which makes diagnosing easier.

With RhinoMocks, it gives clearer messages about expectation
failures.
Here's the message without Shouldly:

Rhino.Mocks.Exceptions.ExpectationViolationException:
IContestant.PlayGame("Shouldly"); Expected #1, Actual #0

Shouldly's message:

Expected:
IContestant.PlayGame("Shouldly");
Recorded:
IContestant.PlayGame("Debugging");
IContestant.PlayGame("Logging");
IContestant.PlayGame("Drinking coffee");
IContestant.PlayGame("Commenting out test");

More at: http://snappyco.de/articles/2010-02-02-shouldly
Source at: http://github.com/snappycode/shouldly

Hit me up for any questions, feature requests or even better fork it!

[Charlie Poole]

Hi Dave,

It's cool...

but how come you aren't involved in helping us get this sort of
syntax into NUnit? *

Charlie

* which we are doing, but maybe it will be less cool without you!

> --
> You received this message because you are subscribed to the
> Google Groups "Seattle area Alt.Net" group.
> To post to this group, send email to altnet...@googlegroups.com.
> To unsubscribe from this group, send email to
> altnetseattl...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/altnetseattle?hl=en.
>
>

[Dave Newman]

Hey Charlie,

I'm happy to contribute patches to NUnit! At the moment I'm experimenting with syntax and I make some assumptions that I'm not sure the majority will agree with. 

It's definitely less flexible than the current Assert.That and involves mixing methods into Object something like:

public static void ShouldBe(this object actual, object expected)

{

    Assert.That(actual, Is.EqualTo(expected));

}

If you're happy to have that in nUnit I can create a patch with:

ShouldBe
ShouldNotBe

ShouldBeGreaterThan
ShouldBeLessThan
ShouldContain
ShouldNotContain

etc.

On the contextual error messages though, I'm pretty sure people won't want that in nUnit core as it involves parsing the code file that was compiled into the running test. In the future I'll also be evaluating variables on the stack. These kinds of things are only for those with strong stomachs and loose morals!

Just as an aside, contributing to nUnit would be much easier if it was on github ; )

--
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Dave Newman | @whatupdave | http://snappyco.de

[David Foley]

Re: the simple NUnit wrappers, I agree that this is cool, but I don't think it's new.

http://code.google.com/p/specunit-net/source/browse/trunk/src/SpecUnit/SpecificationExtensions.cs 

Personally, these extension methods are the kinds of things I usually write piecemeal as I work on a project, to support the types of assertions that are relevant to the problem domain.

Re: the Rhino Mocks extension: That's pretty cool. That should go into Rhino Mocks core IMO.

[Erick Thompson]

Dave,

 

Instead of parsing source code, is there any reason you would use reflect to inspect the stack? That should give you the funciton entry point without having to need access to the source code (however, the source code includes goodies like comments and such).

 

Erick

[David Foley]

On parsing the source code, or even reflecting... I'm curious: what is the benefit of this? 

It seems like "expected Foo but got Bar", combined with the name of the test that is failing, is sufficient.

[Dave Newman]

Reflection won't actually get me the name of the method/property that ShouldBe was invoked upon. 

eg.

    zombie.Mood.ShouldBe("hungry")

The stack trace won't have the instance name "zombie" or the property name "Mood". If I could get to the AST that would be fantastic but alas I think I'm stuck with parsing the source code to get the info I need.

[Dave Newman]

It's just more context. Especially when you get a failing test in CI if you've got:

    Expected "Hungry" but was "Shuffling aimlessly"

then you have to do a bit of investigation (maybe you have several "asserts" in one test, shocking I know :)

but if you had

zombie.Mood should be "Hungry" but was "Shuffling aimlessly"

you can narrow in a bit quicker. Also the ability to do easy IEnumerable asserts:

    myArray.ShouldBe(1,2,3)

this will give you:

    myArray should be [1,2,3] but was [2,3,4]

which is nice   

[Shawn Neal]

You could get access to the AST if you used an Expression<T>, but it doesn't allow for the easier reading Should() extension method syntax.

[Test]

public void Shoudly()

{

    int points = 1445;

    // Message is: Expected: 333  But was:  1445

    //Assert.That(() => points, Is.EqualTo(333));

    // Message is: points should be <equal 333> but was 1445

    AssertEx.That(() => points, Is.EqualTo(333));

}

public static class AssertEx

{

    public static void That<T>(Expression<Func<T>> expression, EqualConstraint equalTo)

    {

        T val = expression.Compile().Invoke();

        if (!equalTo.Matches(val))

        {

            throw new Exception(

                string.Format(

                    "{0} should be {1} but was {2}",

                    GetParamName(expression),

                    equalTo,

                    val));                

        }

    }

    private static string GetParamName<T>(Expression<Func<T>> expression)

    {

        var memberExpression = expression.Body as MemberExpression;

        if (memberExpression != null)

        {

            return memberExpression.Member.Name;

        }

        return "";

    }

}

[Dave Newman]

Thanks for that Shawn. I did originally play around with Expression<T>, but as you say you trade off the test syntax for the message. The trade off with my approach is if the source isn't around, you don't get a cool error message. That's an edge case for me.

[David Foley]

OK, this is somewhat curmudgeonly on my part, I admit, but:

It seems like more benefit is to be found in helping developers write single-assertion tests.

Enabling the writing of tests that can fail for more than one reason seems counter-productive.

[Dave Newman]

Somewhat curmudgeonly? : )

Changing your syntax from

    Assert.That(zombie.Mood, Is.StringContaining("brains")

to

    zombie.Mood.ShouldContain("brains")

and improving the error message doesn't encourage structuring your test with more or less asserts.

Also, by your logic wouldn't that mean we should remove more information from the current test failing?

eg.

Shouldly: 

Zombie_Tests.OnCreationZombieShouldWantsBrains: failed 

zombie.Mood should contain "brains" but was "shuffling aimlessly"

NUnit: 

Zombie_Tests.OnCreationZombieShouldWantsBrains: failed 

Expected string containing "brains"

FoleyTest:

Zombie_Tests.OnCreationZombieShouldWantBrains: failed 

With FoleyTest you'd definitely not want to put more than one assert in : )

I tell you what I'm hooked on informative messages and I suspect others will be too. I've been able to fix tests a lot faster now than I used to.

Here's my new position on software development: 

"There are no rules" — Dave Newman

[David Foley]

I think you're misunderstanding me... I was just speaking about using the stack frame or parsing source to get more context information.

I use similar .ShouldXXX extension methods when I write code in .NET.

I'm saying that the type & name of the test should convey enough information to know what failed.

As far as single-assertion-per-test, I think that is a sound practice, personally. 

It doesn't mean a single _assert_statement_ in all cases. But, IME, 95% of the time it does.

I would lean toward decoupling the context ("on creation") from the expectations ("should wants brains", presumably among others).

Tests.Zombie.when_initially_created.wants_brains.

Nothing against more information, but it's not necessary when the test is just an assertion.

[Charlie Poole]

Hi Dave,

 

We've been having some lively discussions about what new syntax wrinkles to put into the

next release of NUnit. Simone Busoli has branched a slightly different alternate syntax to

yours. You'd be welcome to join us, of course.

 

As far as ease of contribution, I think you'll find that Launchpad has approximately the

same features as GitHub, in particular allowing you to easily create your own branch.

Of course, it's another environment to learn and a different DVCS to master, and I

understand it takes a bit of effort to get over the hump.

 

However, our mail list works just like this one. :-)

 

Charlie

---

From: altnet...@googlegroups.com [mailto:altnet...@googlegroups.com] On Behalf Of Dave Newman

Sent: Monday, February 08, 2010 10:27 PM
To: altnet...@googlegroups.com
Subject: Re: Shouldly testing library

[Charlie Poole]

Hi Dave,

 

One other thing... I could see the code parsing bit as an addin - those with "strong stomachs

and loose morals" could install it without anyone else needing to worry.

 

Charlie

---

From: altnet...@googlegroups.com [mailto:altnet...@googlegroups.com] On Behalf Of Dave Newman

Sent: Monday, February 08, 2010 10:27 PM
To: altnet...@googlegroups.com
Subject: Re: Shouldly testing library

[Charlie Poole]

Hi All,

 

I have to smile as I read this thread.

 

I spent a few months arguing why I thought the Assert.That syntax would never

work before I finally spiked it and used it to provide myself some experiential data.

In the end, it surprised me by working well, so I implemented it in NUnit and it seems

like most people like it better than what came before.

 

Henry Ford is supposed to have said "If I asked people what they wanted, they would

have asked for a faster horse." I don't know if that's accurate, but it seems like there's

a truth of some sort in the sentiment. 

 

As geeks, we like to debate about this stuff almost as much as we like to code it.

Maybe we need a session where we all try one another's implementations and see

how it actually feels to use them.

 

Charlie

 

---

From: altnet...@googlegroups.com [mailto:altnet...@googlegroups.com] On Behalf Of Dave Newman

Sent: Tuesday, February 09, 2010 7:57 PM

To: altnet...@googlegroups.com
Subject: Re: Shouldly testing library

[Dave Newman]

Haha, thanks for that Charlie. I must admit, I posted this to a few lists and got a mostly negative response. At this point I became convinced I was doing the right thing : )

Plus we've used it internally for 3 months now and we love it.  

Anyhoo I joined the nunit mailing list, so I'll chat over there about what might work in nUnit.