To Core Developers: gallio-user asking if Gallio is dead

[Kobus van der Merwe]

The Core Developer Teams of Gallio and MbUnit

There has been some questions on the gallio-user group whether the project is dead.

I am sure there is still great dependency out there on the Gallio framework (Sonar C# for example, and certainly many private source projects' test automation). Is this the case? Can someone then assist in establishing a new team / handover? I am sure there are many developers out there who would love to see this effort continue. I might be able to some extent. This project had no equal in it's abilities.

Best regards,

Gallio addict

[Andrew Stopford]

Hi,

I'll respond later today however in answer to the question the core team has all but dissolved, I my self just don't have the time (I really wish I did) and yes it's painful to see it drift away.

If folks are willing and able then yes I would love to see a new core team with lots of new energy.

Andy

--
You received this message because you are subscribed to the Google Groups "gallio-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gallio-dev+...@googlegroups.com.
To post to this group, send email to galli...@googlegroups.com.
Visit this group at http://groups.google.com/group/gallio-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Justin Webster]

Andy,

What is involved in leading the core team? What would be the responsibilities, time commitment, etc.? I might potentially be interested, but I'd need some details. Thanks.

[Andrew Stopford]

Hi Justin,

The core team has all but moved on so the first step is finding some folks to join the core team. The next few things that need to happen.

Break up the source, it needs moving to it's own repros the build process is a very complicated chain however. It will let sub projects live on or die however and allow the core team to focus on the core.

Migrate the websites - they won't live forever on Jeff's server. Maybe crowd funding could help pay for a AWS or Azure account?

The only thing to consider in leading the core team is letting the team decide how best to run the project.

If a few folks want to start running the core team then let me know so that SVN access can be assigned etc?

Thanks

Andy

[Wayne Bradney]

I was given commit privilege a couple of years ago when I was working with Grahame on a custom testing framework and Icarus plugin. Since that work completed I've been using the 3.3.1 release and haven't found the need to make any other changes to the Gallio sources, so I haven't really kept up with recent developments. That said, I'd be happy to chip in where I can and where the day job allows...

 

I'd strongly agree that the immediate priorities should be to establish a shared, hosted home for the website (I'd be happy to contribute financially to this), and to simplify the build - I remember that being difficult to get started with originally. In my case I was really only interested in NUnit/MbUnit support so all the other extensions and sub-projects just seemed to complicate life somewhat. I've set up TeamCity continuous build systems a few times in previous lives, so I may be of help with this too, but I'm not sure what the requirements might be for an automated process for a public open-source project such as this.

 

This might be a good opportunity to get off SVN and onto, for instance, a new git repo. [Disclosure: I have a personal pathological hatred of SVN - let's call it a 'tree conflict' :-)]

To unsubscribe from this group and stop receiving emails from it, send an email to gallio-dev+unsubscribe@googlegroups.com.

To post to this group, send email to galli...@googlegroups.com.
Visit this group at http://groups.google.com/group/gallio-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to the Google Groups "gallio-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gallio-dev+unsubscribe@googlegroups.com.

To post to this group, send email to galli...@googlegroups.com.
Visit this group at http://groups.google.com/group/gallio-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Cliff Burger]

We've been asking the same question ourselves in my engineering department.

What makes for a successful open source project? A company can takes it under their wing for a variety of reasons. E.g. the android os drives consumers into Google's advertising wing. MySql can up sell. Opscode makes it's money off chef consulting and hosting.

Others are backed by foundations or funded by donations from users. (Wikipedia).

A few months ago, I discussed whether my company (daptiv) could take it on, at least so that I could use time on it. However, it has little to with our core business and is only one of several frameworks that we use.

Getting a core group with day jobs will be a challenge. I could offer a little time, assuming it would look good on a resume.

I agree with others that have suggested that we greatly simplify gallio. Let's keep the core test framework and a command line runner going first. Everything else can and should be addons.

Definitely git. Github?

A website would be nice, but is not essential. Docs in packages, readmes, and samples. Distribution via NuGet.

[Cliff Burger]

[Greg Young]

I think a real discussion should include what is the value statement of keeping it alive (aside from maintenance). What is the mission statement of the new team?

--
You received this message because you are subscribed to the Google Groups "gallio-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gallio-dev+...@googlegroups.com.
To post to this group, send email to galli...@googlegroups.com.
Visit this group at http://groups.google.com/group/gallio-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

--
Le doute n'est pas une condition agréable, mais la certitude est absurde.

[Charlie Poole]

I hesitate to chime in, given that everyone knows my interest is with a different framework. ;-)

However, I will... I've noticed that periodically asking what the products I work on provide as value is helpful in renewing committment and - maybe more importantly - focusing that committment in the right direction. By asking, I mean both asking oneself and asking the users. In the case of NUnit, if I had only asked myself, I would probably be working on a different set of things than I am - whether for goood or bad, time will tell.

Regarding corporate support - as discussed in another post - it seems to me that testing frameworks are only the core business of companies that sell testing frameworks. Therefore, full sponsorship is probably out of the question. But some level of cooperative sponsorship might be sought. Of course, that ties back to the value statement. What are they supporting and why?

Charlie

[Kobus van der Merwe]

Wow this is an even better reaction than I expected.

It seems there is a lot of interest to continue the effort. I agree mostly with the posts above, in particular the following points:

• Move the code to Git
• Dedicated web host
• Simplify the build (very NB, Gallio simplifies test automation, and the build should reflect that)
  

I also agree with revisiting the value statement. My angle on it is that Gallio provides me with the tools to unify otherwise incompatible test frameworks, and has great test run reports that can be reused in other tools (like ccnet build artifacts, Sonar, etc.). It also enhances my testing workflow in Visual Studio, and Icarus has a great user experience with a clean and modern feel. All of these together is what makes me use the project.

I will have some limited time available from middle April, but I am not familiar enough with the code base to get going. I suggest we work with Andy to establish a new team and find a lead. I don't believe I am the most qualified at the moment to take the lead.

Regards,

 - Kobus

[Cliff Burger]

Speaking of Charlie and Nunit, another approach we could take is to look at the features we love about mbunit and stuff them into nunit. As add-ins if not core features. 

Essential - Extensibility: It is trivial to extend MBUnit using decorators inline in an assembly. Nunits addin model (seperate files to load) is more difficult to write and complicates tooling, versioning, and team development. For example, we use decorators to enable autofaq resolution of test fixture properties.

Important: Reports and output: Gallio's html reports are beautiful, and, can include images and video. Images are very useful when debugging web ui tests. It should be possible to port the HTML reporting to the nunit exe or an add-in. Image logging would probably require that the tests reference a different assembly.

Nice: MBunit has much more powerful data driven testing. Nunit's static data driven tests load all test cases on assembly load. 

Initially, we have moved to mbunit for the simple and native parallel testing. However, executing several threads from one test runner process turned out to have more disadvantages than advantages. 

Things about mbunit that I don't like or don't give a damn about: bloat. We love it because it can do so much (run rspec in iron ruby?), but mixing these this with the core distribution makes it prohibitively to maintain. For example, I don't have VS 2008 installed -- thats the version you need for the VS extensions -- so I can't build mbunit.

On Sunday, March 24, 2013 1:22:15 PM UTC-7, charlie wrote:

I hesitate to chime in, given that everyone knows my interest is with a different framework. ;-)

However, I will... I've noticed that periodically asking what the products I work on provide as value is helpful in renewing committment and - maybe more importantly - focusing that committment in the right direction. By asking, I mean both asking oneself and asking the users. In the case of NUnit, if I had only asked myself, I would probably be working on a different set of things than I am - whether for goood or bad, time will tell.

Regarding corporate support - as discussed in another post - it seems to me that testing frameworks are only the core business of companies that sell testing frameworks. Therefore, full sponsorship is probably out of the question. But some level of cooperative sponsorship might be sought. Of course, that ties back to the value statement. What are they supporting and why?

Charlie

On Sun, Mar 24, 2013 at 1:06 PM, Greg Young <gregor...@gmail.com> wrote:

I think a real discussion should include what is the value statement of keeping it alive (aside from maintenance). What is the mission statement of the new team?

On Sunday, March 24, 2013, Cliff Burger wrote:

We've been asking the same question ourselves in my engineering department.

What makes for a successful open source project? A company can takes it under their wing for a variety of reasons. E.g. the android os drives consumers into Google's advertising wing. MySql can up sell. Opscode makes it's money off chef consulting and hosting.

Others are backed by foundations or funded by donations from users. (Wikipedia).

A few months ago, I discussed whether my company (daptiv) could take it on, at least so that I could use time on it. However, it has little to with our core business and is only one of several frameworks that we use.

Getting a core group with day jobs will be a challenge. I could offer a little time, assuming it would look good on a resume.

 I agree with others that have suggested that we greatly simplify gallio. Let's keep the core test framework and a command line runner going first. Everything else can and should be addons.

Definitely git. Github?

A website would be nice, but is not essential. Docs in packages, readmes, and samples. Distribution via NuGet.

--
You received this message because you are subscribed to the Google Groups "gallio-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gallio-dev+unsubscribe@googlegroups.com.

To post to this group, send email to galli...@googlegroups.com.
Visit this group at http://groups.google.com/group/gallio-dev?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Charlie Poole]

Hi Cliff,

On Mon, Mar 25, 2013 at 3:12 PM, Cliff Burger <cliff....@gmail.com> wrote:

Speaking of Charlie and Nunit, another approach we could take is to look at the features we love about mbunit and stuff them into nunit. As add-ins if not core features. 

I guess I wasn't sure if you were talking about just Gallio on this thread, or if mbUnit was also thought to be dead. NUnit has been called dead several times, but has managed to revive, so maybe you can do that too. OTOH, I would welcome good ideas and contributors for NUnit too.

I don't recommend addins. For all the reasons you list below, they are essentially a dead end in NUnit. NUnit 3.0 will support extension of the framework primarily by defining attributes. If there is any addin model, it will be different from what we now have and operate at a higher level - perhaps in the Gui or console runners.

Essential - Extensibility: It is trivial to extend MBUnit using decorators inline in an assembly. Nunits addin model (seperate files to load) is more difficult to write and complicates tooling, versioning, and team development. For example, we use decorators to enable autofaq resolution of test fixture properties.

That's essentially the NUnit 3.0 model, which is also now used in NUnitLite. Attributes operate on tests, decorate them and in some cases are involved in running them. The rough outline is there but it's still under development and thee's room to make changes to the model.
 

Important: Reports and output: Gallio's html reports are beautiful, and, can include images and video. Images are very useful when debugging web ui tests. It should be possible to port the HTML reporting to the nunit exe or an add-in. Image logging would probably require that the tests reference a different assembly.

The intent of the NUnit architecture is that reporting should be based on the test (XML) output and not integrated in the framework at all. That's pretty much how Gallio works too, IIRC. Since the general direction of 3.0 is to disaggregate NUnit, I wouldn't want to add sophisticated reporting to the runners. Anyway, since there are multiple runners, it's better to have reporting separate. IMO, that's an important part of Gallio to save.

Nice: MBunit has much more powerful data driven testing. Nunit's static data driven tests load all test cases on assembly load. 

Planned for 3.0. Needs further definitional work as well as implementation. One path is to have both "static" (they aren't really, but it's the best word I can think of) tests and "dynamic" tests, or if a new kind of tests should replace them all. Since NUnit parameterizes classes as well as methods, there could be a need for work on both levels.
 

Initially, we have moved to mbunit for the simple and native parallel testing. However, executing several threads from one test runner process turned out to have more disadvantages than advantages. 

This exists in my private builds of 3.0. I agree that it presents problems, especially when tests are not perfectly independent, which is why I haven't released it yet. But we have a good start at it.

NUnit also runs async tests (void, Task and Task<>) - I'm not sure if mbUnit does that.

 

Things about mbunit that I don't like or don't give a damn about: bloat. We love it because it can do so much (run rspec in iron ruby?), but mixing these this with the core distribution makes it prohibitively to maintain. For example, I don't have VS 2008 installed -- thats the version you need for the VS extensions -- so I can't build mbunit.

NUnit has gotten bloated as well. Subdividing it into different modules in 3.0 is the solution I'm working on. We'll see.

You or anyone else with good skills is welcome to contribute to NUnit. Bear in mind that working with NUnit requires a bit of patience: we have much more of a legacy base to support than Gallio had - at least at the start. This slows things down a bit. I'm hoping that separting the framework into separate builds per .NET release will allow faster innovation but to the extent that it introduces conditional compilation it also complicates things. That's just how it is.

But look first at whether you can revive your own project... at least that would be my advice.

Charlie

[Andrew Stopford]

Hi,

This is something that is open to discussion and something that I discussed with Charlie a long time ago but maybe the future of MbUnit lies in NUnit. If many of the features that you find useful in MbUnit/Gallio can also be found (or are planned) for NUnit then maybe a lasting legacy would be to contribute those features to NUnit as v3 grows. By the sounds of what Charlie is saying a great deal of what folks find great in MbUnit is possible in NUnit so it maybe it makese sense that these are developed, again open to discussion.

Andy  

[Espen Albrektsen]

Finally - some activity addressing this question! :-)

I too love MbUnit - I was recently forced to refactor my test suite from MbUnit to NUnit (other tooling no longer supported MbUnit) - and I found there were several pieces missing. IMHO MbUnit really shines in that it is a framework well suited for writing both integration and unit tests. In terms of unit testing, I found that NUnit had all the features I needed - but in terms of integration testing, I found it sorely lacking.

1. [Order] and [DependsOn] attributes are really helpful in terms of writing intelligent integration tests
2. Reporting - I have used MbUnit to execute integration tests running ~12hours - if one test fails, it is crucial to have a test report that can assist in pinpointing the cause of failure - which is often state-depenendent. (Hence integration test)

So moving forward: I like the idea of merging the good stuff from MbUnit into NUnit - but do we think it is feasible? I have not reviewed the NUnit codebase - and only briefly touched the Gallio code base - (Patched the R# test runner)

I have quite a bit of experience with configuring Jenkins (formerly Hudson) - and I have my own integration server in my basement. We could use that - but I am not sure if it will be able to handle the load - it is rather underspeced.

I am more than willing to contribute to keeping MbUnit/Gallio alive - but I do not think I will have the time to be the lead.

Anyone?

[Kobus van der Merwe]

It is good to consider the idea of taking MbUnit features to NUnit, but I am also concerned about the feasibility. The nice features of MbUnit evolved because it was a small and new code base that could move fast. The same will not be possible in NUnit with a larger dependent community and legacy to answer to. It is still worthwhile to investigate though.

We should not neglect reviving Gallio though, which is to me where the real value of these two projects lies.

Regards,

 - Kobus

[Justin Webster]

I think that MbUnit probably cannot be merged with nUnit without losing essential (in my mind) MbUnit functionality and/or major changes to some nUnit design that probably would not be good for some of nUnit's users. The primary reason I chose MbUnit was for it's powerful integration testing features. And I believe that some of those features rely on design decisions which may fundamentally conflict with nUnit design decisions. (I don't think that is necessarily a bad thing. It means that the two combined can satisfy conflicting requirements that might be mutually exclusive with an either/or design.)

I have worked on the MbUnit code, and as I understand it there is (at least) one fundamental difference in the test execution design between the 2 frameworks (someone please correct me if I am wrong):

MbUnit/Gallio has dynamic test 'discovery', such that the results of executing one test can result in 'new' tests being discovered/created. Or in other words, the 'list of tests' that will be executed can be modified throughout the test execution process, such that the total number, types, and names of the tests could change at test execution run time. OTOH, nUnit (as I understand it) and all other unit test frameworks have a phased test execution strategy of "discover tests, create a test execution sequence, execute tests". Once past the initial test discovery phase, no new tests can be dynamically added to the test execution.

While this difference may seem unimportant to many, I think that it is a critical feature for some, as it more closely mimics human testing behavior. As an example, when humans test manually and get a 'suspicious' result, we often repeat the previous test steps and/or run additional (different) test steps that we wouldn't otherwise execute if the result hadn't been suspicious. This is more than simply substituting an 'initial' test result with a different result, it is actually a dynamic change to the contents/sequence of the 'test plan' that occurs at run time during execution of the plan.

In my view, this feature of MbUnit's execution mechanism would eventually evolve to support the ability for Gallio/MbUnit to do something similar to TestLoadBalancer, but without requiring that all tests be discovered statically before the tests are balanced and executed (as TLB currently requires). There are also additional features that I think would be very useful for 'integration-oriented' testing and that would rely on this dynamic execution plan feature.

Someone please correct me if any of my assumptions or assertions above are incorrect. Thanks.

--

[Justin Webster]

Charlie:

I would be very interested in your opinion (specifically) about any fundamental differences (whether in philosophy, vision, design, or implementation) between nUnit (both current and future) vs MbUnit/Gallio. I know that with one or both sides continually evolving this is a perennially recurring question, but I'd like your view of it as it stands currently. Now that (as I understand it) nUnit supports many of the data-driven testing (and other) features that were once major differentiators between nUnit and MbUnit, it would be useful to do a re-evaluation of the differences between the two.

Also, for the past several years, users' questions usually take the form of 'which one of these should I use?'. Given the (aforementioned) increased overlap in functionality between the two, I'd like to see whether and how we could instead promote a strategy of "let's use both nUnit and MbUnit with Gallio as the glue, and here's the recommended strategy for when and how nUnit/MbUnit should be used vs. the other". People don't think of using 'either' a unit test framework OR a mocking framework, they know that both are complementary. I'd like side-by-side use of both MbUnit and nUnit to be seen as a long-term-viable alternative to using 'either' one or the other. To be seen as simply having 2 tools in the toolbox instead of one. But the main problem with that has always been knowing the capabilities and strengths of each tool relative to the other, and in knowing when and how to use each tool.

Further, if we (the combined MbUnit and nUnit core teams) can also identify the similarities AND differences in vision and philosophy between the two frameworks, it would allow us to optimize each one for its own niche without having to unnecessarily duplicate features and efforts.

Everyone's feedback on this idea would be appreciated.

[Charlie Poole]

Hi Espen,

Historically, NUnit specifically targeted unit testing and generally rejected enhancement requests that were aimed at integration testing. There was also a general feeling among the developers that features should not be offered (like ordering of tests) that would "tempt" inexperienced people into bad practices like dependencies of unit tests.

However, this hasn't been true for some years now. We _would_ like to make it possible for newcomers to use NUnit for unit- or micro-testing without being confused by extra facilities that will only harm them but we will (and have) added such features. So features such as ordering and dependencies can be part of it - are even planned for 3.0 - but we'd like to somehow make it clear that they are aimed at integration testing, not unit testing. One way to do that is to have a separate attribute that identifies integration tests, replacing TestFixture. Another possibility is to simpy put things like [Order] in a separate namespace. IOW, I don't think the difference of philosophy is all that great, even though there are shades of difference, resulting from different groups of people being involved.

The NUnit Vision Statement has not been updated for a while, but I think it still pretty much says what we are about... http://nunit.org/wiki/doku.php?id=dev:vision

Regarding dependency specifically, I think there are some important shades of meaning in various ways the term is used. I'd be glad to discuss that with anyone wanting to implement a dependency mechanism.

Regarding "merging good stuff," there is always that nasty warning about observing copyright and licensing requirements. NUnit and MbUnit are each using two different licenses for our older and newer versions - that's four licenses in all. If we were to get started in implementing features from MbUnit in NUnit, we'd have to closely look at the source and target licenses. Alternatively, if the person who wrote the code is still around and wants to re-contribute it to NUnit, then anything  goes. In other cases, it may be necessary to re-develop the feature without recourse to the MbUnit code.

Charlie

--

[Charlie Poole]

Hi Justin,

On Tue, Mar 26, 2013 at 10:18 AM, Justin Webster <justin....@gmail.com> wrote:

Charlie:

I would be very interested in your opinion (specifically) about any fundamental differences (whether in philosophy, vision, design, or implementation) between nUnit (both current and future) vs MbUnit/Gallio. I know that with one or both sides continually evolving this is a perennially recurring question, but I'd like your view of it as it stands currently. Now that (as I understand it) nUnit supports many of the data-driven testing (and other) features that were once major differentiators between nUnit and MbUnit, it would be useful to do a re-evaluation of the differences between the two.

I think it would be cool to develop a side-by-side comparison that was supported by both NUnit and MbUnit developers. Usually, comparisons between frameworks are made by proponents of one package wanting to show that they are better than another - I once did an NUnit vs MSTest comparison of that kind myself. :-) If we had something that we all agreed on as factual, it would be useful in itself as well as a great basis for whatever comes next.

We could start with a simple comparison of visible features - available attributes, for example - but should go deeper than that. For example, NUnit attempts to deal with some (but not all) concerns regarding dependencies by use of a hierarchy of before and after attributes but doesn't allow explicit ordering of tests. So simply saying that MbUnit has [Order] and NUnit doesn't is only part of the story. I'm sure there are examples that go in the opposite direction as well. So we need to not only look at attributes but at why we use them and then determine how - if at all - the corresponding need is met by the other framework.

If we had such a comparison, I'd be glad to host it somewhere, along with an annotation that explained when the comparison had last been reviewed for accuracy by each team. I don't think I've ever seen a comparison like that (i.e. agreed upon by both sides) for any software product.
 

Also, for the past several years, users' questions usually take the form of 'which one of these should I use?'. Given the (aforementioned) increased overlap in functionality between the two, I'd like to see whether and how we could instead promote a strategy of "let's use both nUnit and MbUnit with Gallio as the glue, and here's the recommended strategy for when and how nUnit/MbUnit should be used vs. the other". People don't think of using 'either' a unit test framework OR a mocking framework, they know that both are complementary. I'd like side-by-side use of both MbUnit and nUnit to be seen as a long-term-viable alternative to using 'either' one or the other. To be seen as simply having 2 tools in the toolbox instead of one. But the main problem with that has always been knowing the capabilities and strengths of each tool relative to the other, and in knowing when and how to use each tool.

Fair enough, although it won't keep NUnit from continually trying to add features that compete with MbUnit's nor, I suspect, you with ours. :-)

 

Further, if we (the combined MbUnit and nUnit core teams) can also identify the similarities AND differences in vision and philosophy between the two frameworks, it would allow us to optimize each one for its own niche without having to unnecessarily duplicate features and efforts.

Possibly... but I have a bit of a negative reaction that someone should decide the appropriate niche for each and expect them to keep to that niche. It's in the nature of things that we should each try to fill whatever space we can reasonably fill. If market forces or access to resources limit us, that's natural too. But deciding in advance smacks to some extent of the approach taken by certain large companies, with which we each compete, but which we don't necessarily want to emulate.

That's not to say I would never agree with such an approach AFTER we determined the existing differences of vision and philosopy, only that I'm a little hesitant about it.

Charlie

[Justin Webster]

On Tue, Mar 26, 2013 at 11:55 AM, Charlie Poole <nuni...@gmail.com> wrote:

Hi Justin,

On Tue, Mar 26, 2013 at 10:18 AM, Justin Webster <justin....@gmail.com> wrote:

Charlie:

I would be very interested in your opinion (specifically) about any fundamental differences (whether in philosophy, vision, design, or implementation) between nUnit (both current and future) vs MbUnit/Gallio. I know that with one or both sides continually evolving this is a perennially recurring question, but I'd like your view of it as it stands currently. Now that (as I understand it) nUnit supports many of the data-driven testing (and other) features that were once major differentiators between nUnit and MbUnit, it would be useful to do a re-evaluation of the differences between the two.

I think it would be cool to develop a side-by-side comparison that was supported by both NUnit and MbUnit developers. Usually, comparisons between frameworks are made by proponents of one package wanting to show that they are better than another - I once did an NUnit vs MSTest comparison of that kind myself. :-) If we had something that we all agreed on as factual, it would be useful in itself as well as a great basis for whatever comes next.

We could start with a simple comparison of visible features - available attributes, for example - but should go deeper than that. For example, NUnit attempts to deal with some (but not all) concerns regarding dependencies by use of a hierarchy of before and after attributes but doesn't allow explicit ordering of tests. So simply saying that MbUnit has [Order] and NUnit doesn't is only part of the story. I'm sure there are examples that go in the opposite direction as well. So we need to not only look at attributes but at why we use them and then determine how - if at all - the corresponding need is met by the other framework.

I was actually thinking of maybe approaching it sort of like a set of user stories. For example:
IF I want to get my test methods to run in a specific order...
THEN (IF I am using nUnit) I would use the ... attribute OR I would ...
THEN (IF I am using MbUnit) I would ...

That way both sides could contribute 'requirements' or features, but individual sides would be responsible for showing their own framework in the best light. I guess it could be kind of like a voter's pamphlet too. With each side giving it's own 'PRO' argument and the other side 'if desired' adding a rebuttal. Personally, all I'm interested in (as a test framework consumer/user) is obtaining unbiased info about all the alternatives so I can make my own decision. So I'd be happy with whatever format would best meet the need of presenting decision makers with a minimally-biased (or at minimum bias from both sides to counter balance the biases) comparison of features and capabilities, etc.

If we had such a comparison, I'd be glad to host it somewhere, along with an annotation that explained when the comparison had last been reviewed for accuracy by each team. I don't think I've ever seen a comparison like that (i.e. agreed upon by both sides) for any software product.
 

Also, for the past several years, users' questions usually take the form of 'which one of these should I use?'. Given the (aforementioned) increased overlap in functionality between the two, I'd like to see whether and how we could instead promote a strategy of "let's use both nUnit and MbUnit with Gallio as the glue, and here's the recommended strategy for when and how nUnit/MbUnit should be used vs. the other". People don't think of using 'either' a unit test framework OR a mocking framework, they know that both are complementary. I'd like side-by-side use of both MbUnit and nUnit to be seen as a long-term-viable alternative to using 'either' one or the other. To be seen as simply having 2 tools in the toolbox instead of one. But the main problem with that has always been knowing the capabilities and strengths of each tool relative to the other, and in knowing when and how to use each tool.

Fair enough, although it won't keep NUnit from continually trying to add features that compete with MbUnit's nor, I suspect, you with ours. :-)

 

Further, if we (the combined MbUnit and nUnit core teams) can also identify the similarities AND differences in vision and philosophy between the two frameworks, it would allow us to optimize each one for its own niche without having to unnecessarily duplicate features and efforts.

Possibly... but I have a bit of a negative reaction that someone should decide the appropriate niche for each and expect them to keep to that niche. It's in the nature of things that we should each try to fill whatever space we can reasonably fill. If market forces or access to resources limit us, that's natural too. But deciding in advance smacks to some extent of the approach taken by certain large companies, with which we each compete, but which we don't necessarily want to emulate.

I was not suggesting that we 'divvy up' the features between the two frameworks. I was mainly suggesting that both teams be more aware of each product's mission/vision. In the past when nUnit was semi-anti-integration testing features, that was a definite philosophical difference between the two products, and one that would help both developers and users make decisions if they were aware of it. As as example, if the MbUnit developers know that nUnit already has great support for a specific feature that has been requested as an enhancement of MbUnit, then the decisions about 'should we do it', and 'how should we do it' may change. Just by being aware that 'the other product' already has support for such a feature, the devs might decide to simply recommend use of that other product in that scenario. OR they might make a different decision. But the important thing (IMO) is that both teams are more aware of the true differences (and similarities) between both products.

That's not to say I would never agree with such an approach AFTER we determined the existing differences of vision and philosopy, only that I'm a little hesitant about it.

Also, if after attempting to determine the differences between each product's philosophy, few or zero major differences are found, then that conclusion in and of itself may be sufficient reason to further investigate a 'merge' of the two products. So we could stop 'competing' and spending time and effort to produce essentially the same (free) product twice and instead combine forces to achieve more together.
 

Charlie

[Charlie Poole]

Hi Justin,

Thanks for your thoughtful response.

On Tue, Mar 26, 2013 at 2:01 PM, Justin Webster <justin....@gmail.com> wrote:

On Tue, Mar 26, 2013 at 11:55 AM, Charlie Poole <nuni...@gmail.com> wrote:

Hi Justin,

On Tue, Mar 26, 2013 at 10:18 AM, Justin Webster <justin....@gmail.com> wrote:

Charlie:

I would be very interested in your opinion (specifically) about any fundamental differences (whether in philosophy, vision, design, or implementation) between nUnit (both current and future) vs MbUnit/Gallio. I know that with one or both sides continually evolving this is a perennially recurring question, but I'd like your view of it as it stands currently. Now that (as I understand it) nUnit supports many of the data-driven testing (and other) features that were once major differentiators between nUnit and MbUnit, it would be useful to do a re-evaluation of the differences between the two.

I think it would be cool to develop a side-by-side comparison that was supported by both NUnit and MbUnit developers. Usually, comparisons between frameworks are made by proponents of one package wanting to show that they are better than another - I once did an NUnit vs MSTest comparison of that kind myself. :-) If we had something that we all agreed on as factual, it would be useful in itself as well as a great basis for whatever comes next.

We could start with a simple comparison of visible features - available attributes, for example - but should go deeper than that. For example, NUnit attempts to deal with some (but not all) concerns regarding dependencies by use of a hierarchy of before and after attributes but doesn't allow explicit ordering of tests. So simply saying that MbUnit has [Order] and NUnit doesn't is only part of the story. I'm sure there are examples that go in the opposite direction as well. So we need to not only look at attributes but at why we use them and then determine how - if at all - the corresponding need is met by the other framework.

I was actually thinking of maybe approaching it sort of like a set of user stories. For example:
IF I want to get my test methods to run in a specific order...
THEN (IF I am using nUnit) I would use the ... attribute OR I would ...
THEN (IF I am using MbUnit) I would ...

That way both sides could contribute 'requirements' or features, but individual sides would be responsible for showing their own framework in the best light. I guess it could be kind of like a voter's pamphlet too. With each side giving it's own 'PRO' argument and the other side 'if desired' adding a rebuttal. Personally, all I'm interested in (as a test framework consumer/user) is obtaining unbiased info about all the alternatives so I can make my own decision. So I'd be happy with whatever format would best meet the need of presenting decision makers with a minimally-biased (or at minimum bias from both sides to counter balance the biases) comparison of features and capabilities, etc.

The voter's pamphlet is a more competitive model than I was hoping for. I think that we could actually create something in common if we stick to straight facts about what each can do. It's hard to introduce the reasons for using a feature (your IF parts) without getting into differences of opinion, but I'd be willing to try.

If we had such a comparison, I'd be glad to host it somewhere, along with an annotation that explained when the comparison had last been reviewed for accuracy by each team. I don't think I've ever seen a comparison like that (i.e. agreed upon by both sides) for any software product.
 

Also, for the past several years, users' questions usually take the form of 'which one of these should I use?'. Given the (aforementioned) increased overlap in functionality between the two, I'd like to see whether and how we could instead promote a strategy of "let's use both nUnit and MbUnit with Gallio as the glue, and here's the recommended strategy for when and how nUnit/MbUnit should be used vs. the other". People don't think of using 'either' a unit test framework OR a mocking framework, they know that both are complementary. I'd like side-by-side use of both MbUnit and nUnit to be seen as a long-term-viable alternative to using 'either' one or the other. To be seen as simply having 2 tools in the toolbox instead of one. But the main problem with that has always been knowing the capabilities and strengths of each tool relative to the other, and in knowing when and how to use each tool.

Fair enough, although it won't keep NUnit from continually trying to add features that compete with MbUnit's nor, I suspect, you with ours. :-)

 

Further, if we (the combined MbUnit and nUnit core teams) can also identify the similarities AND differences in vision and philosophy between the two frameworks, it would allow us to optimize each one for its own niche without having to unnecessarily duplicate features and efforts.

Possibly... but I have a bit of a negative reaction that someone should decide the appropriate niche for each and expect them to keep to that niche. It's in the nature of things that we should each try to fill whatever space we can reasonably fill. If market forces or access to resources limit us, that's natural too. But deciding in advance smacks to some extent of the approach taken by certain large companies, with which we each compete, but which we don't necessarily want to emulate.

I was not suggesting that we 'divvy up' the features between the two frameworks. I was mainly suggesting that both teams be more aware of each product's mission/vision. In the past when nUnit was semi-anti-integration testing features, that was a definite philosophical difference between the two products, and one that would help both developers and users make decisions if they were aware of it. As as example, if the MbUnit developers know that nUnit already has great support for a specific feature that has been requested as an enhancement of MbUnit, then the decisions about 'should we do it', and 'how should we do it' may change. Just by being aware that 'the other product' already has support for such a feature, the devs might decide to simply recommend use of that other product in that scenario. OR they might make a different decision. But the important thing (IMO) is that both teams are more aware of the true differences (and similarities) between both products.

I see. Good point.

That's not to say I would never agree with such an approach AFTER we determined the existing differences of vision and philosopy, only that I'm a little hesitant about it.

Also, if after attempting to determine the differences between each product's philosophy, few or zero major differences are found, then that conclusion in and of itself may be sufficient reason to further investigate a 'merge' of the two products. So we could stop 'competing' and spending time and effort to produce essentially the same (free) product twice and instead combine forces to achieve more together.

Right... if it works out that way. If not, then they aren't the same - or at least not trying to be the same - and it's not a duplication at all.

I would also add that philosophy and features are two different things. At least, we can address them separately.

Charlie

 

Charlie