Re: Choosing between xUnit, mbUnit, and NUnit

[x97mdr]

Personally, I like NUnit because more external tools understand the XML format it can produce.  Many more plugins seem to support NUnit over xUnit, mbUnit (for example, Hudson/Jenkins had greater support for capturing test trends with NUnit initally)

On Thursday, December 6, 2012 6:16:43 PM UTC-5, Oran Dennison wrote:

I'm new to SpecFlow and trying to choose a test provider.  My preferences are xUnit, mbUnit, and NUnit, in that order.

The SpecFlow support matrix for unit test providers makes it appear that xUnit has some SpecFlow limitations (categories, inconclusive) that mbUnit and NUnit don't have.  Other resources indicate that NUnit was the only one with good report generation, but that was a few years ago.

So what do people recommend?  Are xUnit, mbUnit, and NUnit all equals now when it comes to SpecFlow feature support, or is it best to stick with NUnit to get the most out of SpecFlow?

[Darren Cauthon]

I'd use NUnit, for two reasons:

1.)  It works out-of-the-box, and

2.)  SpecFlow use doesn't change per test framework, as it sits on top of the test framework.

Seeing people knowingly choose frameworks with issues and then complain about those issues is one of those matters that I just don't understand.  Devs prefer a framework for whatever reason (though usually because it's just what they've used in the past), so they try to force Specflow to use that framework almost like a zombie.  "Muuuuusssssttttt ussssseeeeeeeee MSSSSSSSSSSSSTeeeeeeeeeeesssssttttt."  But what would change if you used NUnit instead of MSTest, or the others?  Other than *perhaps* the assertions, the code in SpecFlow will work exactly the same.  In reality, we're discussing what is an invisible, behind-the-scenes tool... so why not use the one (or two) that works?

I understand there are things like build processes, tooling concerns, or arbitrary architect demands that sometimes force us into picking one.  These things are very disappointing, but I think sometimes it would be easier to overcome them than try to take a tool down the sad path.  There is a happy, care-free path with SpecFlow, and it's with NUnit.  Perhaps the others work well, too, but years ago when I saw that NUnit worked, I stopped looking. :)

Darren

[Pedro G. Dias]

Darren, I'd be careful about throwing crap at frameworks other than the one you prefer yourself. There's something about that type of posts that removes the legitimacy of whatever else you have to say - Roy Osherove, for example, lost al lot of followers when he started to badmouth competing IoC containers over the one his x-employers were developing. He is now freezing his stones off with -17C here in Norway :-)

The only reason for changing one test framework over another is because the one you currently use is limiting your options in some way, or because another test framework offers you a chance to save a serious amount of time. 

In our latest project, we're nearing 3000 unit tests written with MSTest. We follow the SOLID principles for all our work, and we used TestDriven.Net to run MsTest without instrumentation, which made it more than fast enough for our needs (currently a handful of seconds to run 3000 unit tests. You have time to blink, but that's about it :)

For all our testing needs, we've never had use for anything but exception assertions on some tests, as well as setting up a few static "before first" initialisers (ClassInitialize) when doing IO integration tests. There are more advanced uses in all of these frameworks, however, if you're working SOLID, keeping your code DRY and overall keep a sound relation to the code you're writing, Unit tests nver get to get more complex than a simple setup and even simpler assertion.

nUnit used to be our preference due to its lack of instrumentation. What slowed Mstest down back in VS2008 and later VS2010, was its instrumentaition feature, keeping a record of all data about every single testrun on a new folder.. That was easilly solved by using TestDriven.Net, which ran MsTest without all the instrumentation, but with VS2012, MsTest is fully on par, as you have to opt in with all those things now, MsTest execution is fast again, thus, the argument to use it falls under "MsTest aint broken anymore, no reason to fix it"

In my line of work, I often come across the need to introduce TDD to developers. Doing so with MsTest goes a long way in assuring the devs about two things: 

1) They do not need to install anything in order to start working with TDD

2) If Microsoft has a framework for it, then we can call to get support on it when we're stuck

Of course, nobody ever calls Microsoft for support with MsTest, but the fact that you CAN is often an icebreaker for introducing new methodologies.

As far as the integration with SpecFlow: 

MsTest is supported by SpecFlow .It is also advertised as a supported framework for SpecFlow. Anything that is NOT working against SpecFlow+MsTest that works with nUnit can therefore be tagged as a bug. There is no conceptual difference in those two frameworks, nor should there be. 

[Oran Dennison]

Thank you all for your input.  Looks like my post unintentionally served as flamebait for the age-old unit test framework debates. Sorry about that!  I was specifically interested in the pros and cons that are unique to SpecFlow.

Here's what I've learned over the past week or so.

At a high level, the unit test framework shouldn't matter.  As Marcus says, in SpecFlow a test runner is just glue.  True, but there are differences in the details.

The following SpecFlow + unit test framework NuGet packages exist: SpecFlow.NUnit and SpecFlow.xUnit.  Of course this is just a convenience.  You can independently install NuGet packages for other unit tests frameworks supported by SpecFlow.

Specflow.exe is only able to generate execution reports for NUnit and MSTest.  The current list of supported execution reports can be found by running "specflow.exe help".  You can also check the documentation for details.

Per this post, SpecFlow's NUnit execution reports supposedly don't have the same show/hide functionality to display the individual steps within a scenario.  SpecFlow's MSTest execution reports have this functionality.  I noticed this as well when I first generated an NUnit execution report, but later when I had more feature files, this show/hide functionality suddenly appeared in my NUnit execution reports.  Perhaps there's a bug in the NUnit report that doesn't exist in the MSTest report.

So, I was happily spec'ing along with SpecFlow+NUnit when I decided to investigate how I could add screenshots to my execution reports.  I found a few questions about this but nobody saying "this is how I did it."

This is where the philosophical differences between unit testing and user acceptance testing manifest themselves in framework design choices.  NUnit doesn't appear to have a way to add arbitrary structured data to the test results, although they are open to adding it if someone writes a spec.  xUnit is philosophically opposed to this, saying something like "if you want to do this, you're probably doing it wrong." (agreed, but only in the context of unit testing, not user acceptance testing)  MbUnit supports adding all sorts of custom data with its TestLog class.  MSTest supports this with TestContext.AddResultFile.

So, given all of this information, I'm now considering switching from NUnit to MSTest because it's the only option that will allow me to generate SpecFlow execution reports that include screenshots (after modifying the XSLT).  Due to all the MSTest bashing when it was first released, it wasn't even on my original list of possible options, but it surprisingly looks like the best tool for the job, at least in this context. :-)