Unit testing frameworks

[Herb]

Starter topic for unit testing frameworks. If it turns out to be
easier or more productive to conduct this elsewhere (Wave?), I'm all
for it.

Offhand I can think of the following sub tasks:
1) Info gathering - what's out there that people have specific
experience with (good/bad/ugly)
2) Comparison / evaluation framework - what specific features do we
really need to make sure we're getting
3) Discussions / flamewars & R&D
4) Semi-formal presentation back to the rest of the group

Frankly, I wouldn't expect this to be a huge undertaking, but I
suppose it's going to largely depend on the number of participants.

Step 0 is to probably agree on what our tasks really should consist
of....

--
Herb.

[Jader Dias]

Jon Skeet has suggested NUnit. I prefer MSTest, but it requires an
expensive version of Visual Studio and runs only on Windows.

+ 1 for NUnit + Rhino Mocks

[Tamas Czinege]

2009/11/6 Jader Dias <jad...@gmail.com>

Jon Skeet has suggested NUnit. I prefer MSTest, but it requires an
expensive version of Visual Studio and runs only on Windows.

+ 1 for NUnit + Rhino Mocks

I'm not sure if we could get MSTest working with Mono.

[stuartg]

To be perfectly frank and honest, I'm not exactly sure that there is
much of an argument and/or discussion to be had, NUnit is pretty much
the de-facto standard unit testing tool for C#.

See this question on SO: http://stackoverflow.com/questions/127687/favorite-net-unit-testing-framework,
NUnit comes out on top, it's tried, tested and used by thousands of
developers around the world, in hundreds of projects.

There is also this comparison in the xUnit project:
http://xunit.codeplex.com/wikipage?title=Comparisons&ProjectName=xunit

The biggest appeal behind using MSTest is it's really very nice
integration with Visual Studio, I was hoping in 2010 they'd allow a
method for plugging in a different framework, but alas, no. Other than
it's IDE integration, I personally don't think that it offers very
much, at least over what NUnit offers, and the biggest argument
against it is: Do you want to use a unit testing framework that's only
updated when new versions of Visual Studio are release?

+1 for NUnit and Moq

[Brandon]

+1 for NUnit. As has been stated above, it's essentially the de-facto
standard for unit testing in .NET. Pretty much everyone should already
be familiar with it, and if not, it's easy enough to pick up if you're
familiar with other unit testing frameworks.

On Nov 6, 2:17 pm, stuartg <sgras...@gmail.com> wrote:
> To be perfectly frank and honest, I'm not exactly sure that there is
> much of an argument and/or discussion to be had, NUnit is pretty much
> the de-facto standard unit testing tool for C#.
>

> See this question on SO:http://stackoverflow.com/questions/127687/favorite-net-unit-testing-f...,

[Herb]

Well then. I didn't want to jump in with saying that I luvs me my
NUnit and color the discussion.... but ok.

I *think* that it works fine with Mono, but I haven't tried, nor do I
personally know anyone who has.

I agree that staying away from Visual Studio only solutions is for the
best. The only question I had was with xUnit, but I don't have strong
feelings for it. Thanks for the links Stuart - interesting reading
from the StackOverflow group. I may have to spring for Resharper;
seems to me I used to have a license for it. Hmm.

Does anyone have strong feelings *against* nUnit? Or are we
considering this conversation closed?

Herb.

[Jader Dias]

Case closed for me.

the MSTest Visual Studio integration gives it a very good feature
called Unit Test Wizard, that speeds up a little the coding effort.
I wonder if Monodevelop will have a similar feature.

[Herb]

While we're here - someone (Jader Dias) suggested Rhino Mocks. We may
as well open the floor for discussion here too.

1) Where in particular have you found value in Mock frameworks? My
experience here is much more limited - never in a real, production
project, unlike NUnit.
2) Which one? I'm starting my reading with:
http://stackoverflow.com/questions/642620/what-should-i-consider-when-choosing-a-mocking-framework-for-net
http://www.clariusconsulting.net/blogs/kzu/archive/2008/03/17/WhydoweneedyetanotherNETmockingframework.aspx
http://stackoverflow.com/questions/37359/what-c-mocking-framework-to-use
http://haacked.com/archive/2008/03/23/comparing-moq-to-rhino-mocks.aspx

It looks like .NET platform choice might play a larger role here; MoQ
is only for C# 3.0+ (.NET 3.5), apparently.

Good weekend all.

Herb.

[Eric Sterling]

I found RhinoMock to be rather confusing to use. I know it's powerful, but
it took me a while to wrap my head around how it works. MOQ on the other
hand, seemed very natural to me.

Sooo...I'm ok with either, but I'd prefer MOQ.

~eric

[John Teague]

I would like to suggest Machine Specifications (mspec) for the unit
testing framework. It may look funny at first, but once you get used
to it is a really natural way to describe tests in a bdd style. Aside
from the BDD features of the framework, another feature that would be
very useful is the ability to define test stubs without needing to
implement them. That way we can create the full test suite from the
Joda framework ahead of time and we'll know how far along we are in
fully meeting the specification.

On Nov 6, 8:32 pm, "Eric Sterling" <e.sterl...@comcast.net> wrote:
> I found RhinoMock to be rather confusing to use. I know it's powerful, but
> it took me a while to wrap my head around how it works. MOQ on the other
> hand, seemed very natural to me.
>
> Sooo...I'm ok with either, but I'd prefer MOQ.
>
> ~eric
>
> -----Original Message-----
> From: noda...@googlegroups.com [mailto:noda...@googlegroups.com] On
>
> Behalf Of Herb
> Sent: Friday, November 06, 2009 6:27 PM
> To: Noda Time
> Subject: Re: Unit testing frameworks
>
> While we're here - someone (Jader Dias) suggested Rhino Mocks. We may
> as well open the floor for discussion here too.
>
> 1) Where in particular have you found value in Mock frameworks? My
> experience here is much more limited - never in a real, production
> project, unlike NUnit.

> 2) Which one? I'm starting my reading with:http://stackoverflow.com/questions/642620/what-should-i-consider-when...
> ng-a-mocking-framework-for-nethttp://www.clariusconsulting.net/blogs/kzu/archive/2008/03/17/Whydowe...
> anotherNETmockingframework.aspxhttp://stackoverflow.com/questions/37359/what-c-mocking-framework-to-usehttp://haacked.com/archive/2008/03/23/comparing-moq-to-rhino-mocks.aspx

[Richard Nienaber]

+1 for NUnit

The fact that NUnit is so ubiquitous is what I find compelling. I'm
open to trying other frameworks but it seems the differences between
them are so minimal that we may as well go with the commonly used one.

+1 Moq

I've used RhinoMocks in the past which seemed to require a lot of
ceremony. I currently use Moq at work and it seems pretty
straightforward. I would suggest TypeMock because it allows you to
mock out concrete methods, but it's a non open source, paid for
product. If there's an alternative that does this it would be nice to
go with that.

Having said that, a cursory look at the joda-time java sources seems
to indicate that classic mocking (e.g. they have objects like MockZone
for the express purpose of testing) is used. It may not be necessary
to actual use a mocking frameworkto begin with.

Richard

> > anotherNETmockingframework.aspxhttp://stackoverflow.com/questions/37359/what-c-mocking-framework-to-...

[stuartg]

On Nov 7, 2:27 am, Herb <spoilsportmot...@gmail.com> wrote:
> It looks like .NET platform choice might play a larger role here; MoQ
> is only for C# 3.0+ (.NET 3.5), apparently.

While this is true, you must not lose sight of the fact that it is
only the library itself that is required to support .NET 2.0.
Personally at work I have two projects were I am restricted to .NET
2.0 in the production code, however I have free reign to use whatever
I like in my unit tests.

I believe a consumer of Noda Time would not need to run the unit test
suite, unless they were contributing a feature or bug fix.
Consequently I think we can be perfectly OK in using a mocking
framework which is .NET 3.5 specific. Which leads me onto...

My preference and vote is for Moq, it's easy to use, and has concise
and easy to read syntax.

To add to Herb's list of reading:
http://stackoverflow.com/questions/64242/rhino-mocks-typemock-moq-or-nmock-which-one-do-you-use-and-why

[Martinho Fernandes]

> I would like to suggest Machine Specifications (mspec) for the unit
> testing framework. It may look funny at first, but once you get used
> to it is a really natural way to describe tests in a bdd style.

I've looked into MSpec before and I liked it. But since JodaTime tests
are written in jUnit I think we'll be better off with a similar
framework. I'm starting to like xUnit.net, but I've worked with NUnit
and I like it as well. I think we can say we'll go with NUnit. But now
we have another potato in our hands: which NUnit version? They're are
at least supported versions, and some are "runner-incompatible" with
each other. I had a few problems with that in the past, mainly because
I had to install both 2.2 and 2.4 to work on projects that did not
include the tools with the sources under version control.
The latest (2.5) is probably the way to go, and I can't think why we
wouldn't want that.

I would also vote for Moq as the mock framework, **IF** we ever need
one. If we need only a few mocks/stubs/fakes/whatever, I'm fine with
hand-rolled ones.

[Herb]

I agree with Stuart's comments about the frameworks requirements, with
the caveat that I'd be bummed to be "stuck" on .NET 2.0, and thereby
couldn't run the Mock framework, but I would venture to say that it's
certainly not a deal breaker for me. It would lead to some kind of
funny requirements, that developers or contributors to the project
would need to have at least 3.5 available, but you'd only need 2.0 to
use it. Better than the other way around, at least (if that were even
possible!).

I played a little with MoQ here; it does take a bit to get through the
syntax (I'm still a bit of a tyro with lambda syntax, but getting
there). OTOH, it was a possible to get up and going very quickly...
unlike Rhino which I'm still trying to get my head wrapped around.
Stopping for now, unless there's a really strong push for it.

I also like the suggestion of Mr. Fernendes & Mr. Nienaber - a full
framework might not be necessary or even overkill; I think I'd kind of
alluded to that earlier (w/o benefits of an asbestos suit). OTOH, just
because the Joda project doesn't use it doesn't mean we shouldn't; it
may well be that decent Mocking frameworks were hard to come by in the
Java world when that project started. If we're trying to build a world
class example for others to follow, then I'd argue that we ought to
use the tools necessary to ensure that outcome.

I still like NUnit - and yeah, there's some version warts, last one
I've been using is 2.4.x. If 2.5 is stable (has been since May, I
see), then there's no good reason to not use it that I know of. Anyone
tried it and had issues?

In a similar vein, has anyone done the homework of Moq v3.1 vs 4? 4 is
still in beta, I didn't get it to play with, and I don't know if there
are breaking changes. I'll get the 4.x docs and read unless someone
here has firsthand experience they can share.

Herb.

[stuartg]

On Nov 8, 2:28 am, Herb <spoilsportmot...@gmail.com> wrote:
> It would lead to some kind of
> funny requirements, that developers or contributors to the project
> would need to have at least 3.5 available, but you'd only need 2.0 to
> use it.

Granted, but what developer worth his salt these days doesn't
have .NET 3.5 available to him? I certainly don't think that it is too
onerous a requirement that if you want to contribute to the project
you must have .NET 3.5.

> I also like the suggestion of Mr. Fernendes & Mr. Nienaber - a full
> framework might not be necessary or even overkill; I think I'd kind of
> alluded to that earlier (w/o benefits of an asbestos suit).

I agree, you have to use the right tool for the right job, Roy
Osherove in his "The Art of Unit Testing" makes the point that full
mock frameworks are unnecessary and are sometimes overkill.

> I still like NUnit - and yeah, there's some version warts, last one
> I've been using is 2.4.x. If 2.5 is stable (has been since May, I
> see), then there's no good reason to not use it that I know of. Anyone
> tried it and had issues?

I've been using 2.5 for months, and have yet to run into any issues.

> In a similar vein, has anyone done the homework of Moq v3.1 vs 4? 4 is
> still in beta, I didn't get it to play with, and I don't know if there
> are breaking changes. I'll get the 4.x docs and read unless someone
> here has firsthand experience they can share.

There are no 4.x docs! In fact, there are as yet no published 4.x
improvements, although from the looks of the check-ins (http://
code.google.com/p/moq/source/list) there doesn't appear to have been
any breaking changes.

[Martinho Fernandes]

Still regarding the manual mocking vs. dynamic mocking issue, here's
my pro/con analysis:

Manual mocking Pros:
- Anyone that never used a mock framework can easily understand it
- Easy to mock out protected methods
- No extra dependencies

Manual mocking Cons:
- If you need slightly different behavior for another test, you need
yet another fake class
- May encourage some logic in the fakes (which is not a good idea)
- Hides away the behavior of the fakes
- Requires implementation of *all* abstract (or interface) methods

Dynamic mocking Pros:
- The behavior of the fakes (including the assertions) is kept
together with the test code
- You can mock only what you need

Dynamic mocking Cons:
- Can be a barrier to entry of developers (you need to know the
framework, some people find lambdas confusing, etc)
- Is yet another dependency

I've been diving through the Joda sources and I counted a total of 7
mocks, and most are there to mock just a couple of methods. Because
the interfaces in the library are rather large, this means that there
are a lot of empty (or "return null") methods. I think all this noise
is unnecessary.

Hope this helps making a decision.

[freeranger]

+1 for nunit

re: mocking - I love Rhino, use it every day, and would be happy to go
with that, BUT as Martinho says above, if there is very little use of
mocking in Joda, chances are there would be very little use of mocking
in Noda, so perhaps manual mocking is the way to go?
Maybe we need to defer a decision on this until we actually need some
mocks?

[Bob Gregory]

+1 for xUnit over nUnit - it's a neater, tighter framework with more
options for extensibility when you need them, though nUnit is always a
safe choice.
+1 moq over Rhino, though I'm not sure that NodaTime is likely to need
mocks at all - I'd rather use stubs where we need them, and build for
composability; it cuts down the number of libraries that you need to
ship/keep up to date and it stops you reaching for mocks every time
you need to change some composed behaviour.


-- Bob