Talking about the future and lessons learned.

[Jeff Brown]

I am glad to see there is interest in reviving Gallio and MbUnit.  I am especially tickled that there is interest in sharing ideas and working more closely with Charlie on the evolution of NUnit.

Thanks for taking initiative on this Justin!  I'll definitely help you transition infrastructure and set up whatever you need.

Since people are discussing the future, I thought I might share a few thoughts.

In some ways, I wish MbUnit were much more like NUnitLite.  NUnitLite and xUnit are both very elegant and focused.  MbUnit is more diffuse.  MbUnit has a lot of good ideas but they're not all very accessible.

I think many of the problems commonly cited about Gallio and MbUnit such as complex build dependencies, slow test execution, test harness instability are a result of trying to do too many things with one tool.  The design evolved organically over many releases and it's not as clean and straightforward as it could be.

For example, we started off with tests running in-process just like MbUnit v2.  Then we added app-domain isolation.  Then we added the ability to override runtime options in exe.config.  Then we added process isolation.  Then we added support for crossing x86/x64 architecture boundaries.  Then we added support for DLR languages.  Then we started thinking about how to branch out of .Net altogether and how to run large scale test farms.  Each new feature made things more complex and resulted in a lot of remoting glue and infrastructure.

My point is that with a different separation of concerns, the result would have been quite different and perhaps better.

There are a number of things I would design differently now.  Here are a few ideas.

Common test data format

There is demand for better open source test reporting tools.  I know many integration testers value Gallio's ability to embed rich content and formatting in the reports.  My personal favorite features are structured log formatting and highlighted inline diffs.

Gallio test reports offer many features to help a tester understand why a test failed.  The purpose of the reporting infrastructure is to collect a whole bunch of data during test execution, store it, and present it.

If I were to do this again, I wouldn't attach the reporting infrastructure to the test runner itself.  That's because the people who want to write reporting tools aren't the same as those who want to write test runners and test frameworks.

I would define a common test data format for publishing logs, markup, images, videos, performance counter samples, links to source code, results and other useful information gathered during test enumeration and execution.  The format might just consist of a collection of protobuf messages or some simple JSON (although blobs could pose a problem).

The common test data format shouldn't know or care about any particular test framework or test runner.  It should focus on storing and transporting test data.  It shouldn't hardcode knowledge about how tests are structured.  Better to view tests as a collection of possibly nested steps.

The common test data format should be so simple and have so few dependencies that anyone could implement it with ease.  Imagine that you could run NUnit and ask it to write its results to a file in this format or stream them to a server using a well known protocol.  (Hello Archimedes!)  Imagine you could do this with any test runner or write simple plugins and scripts to transform their native output as required.

Then you almost don't need Gallio anymore because you can build all of the tools you need using the common test data format without special plugins and adapters.

And you're not limited to .Net anymore either.  Write your tests in Python or Go or Lua or Dalvik if you like and use the common tools to run them and generate reports.

Now don't be shy when you design this format.  Make sure it supports streaming and aggregation.  A test harness should be able to write a stream of messages during test runs so if it crashes we still get partial output that we can aggregate with other tests.  And if you want rich reporting then you need to have support for images, videos, blobs, section headings, and whatnot.  Be prepared to handle megabytes of output.

I think Gallio's test runner events (in Gallio.Runner.Events) are a good first step at identifying some of the important phases in test execution to be captured but don't stop there...

Lightweight test harnesses

You can write a skeleton of a test runner in about 50 lines of code.  You just load an assembly, do some reflection, invoke a method to run the tests, and write the results.  Simple.

Anything missing here?  Plenty.  Where's the GUI?  Where's the command to list all available tests?  Where's the reporting?

Well... ok.  So you add all of that stuff.  Now what if your tests don't run on the CLR?  What if your tests need to run as a special user, or on a special machine, or on a different device with a different architecture?  Suddenly you need process isolation and remote control and you're totally screwed.  You can't just load, reflect, run and report anymore.  You've got to fork something off instead and communicate across processes.

So instead of agonizing over all of that, just assume you're going to want process isolation from the get-go.  Build a simple test harness that does nothing more than load code, run tests and emit test data (in that common test data format natch').

Your test harness should not be a program that a user will ever run directly.  It's an internal piece of infrastructure.  It's what the GUI will fork off to run a bunch of tests when the user clicks Run.  It should be extremely lightweight.  You should be able to fork off an instance and have it up and running in milliseconds.  You should even be able to fork a separate instance per test if you like, or fork off ten instances in parallel.  You should be able to launch it on a different machine even.  It should be just that fast.

The input to your test harness should be a stream of tests to run and references to any resources needed to run them (assemblies, configuration parameters, etc.).  The output from your test harness should be a stream of raw test data.  When the job is done, the test harness should just exit and not even bother trying to figure out whether all of the tests passed or failed or even saying goodbye.

Oh yeah, and if your test hangs, just kill the test harness process completely, start a new one and carry on with the next test.  Don't even bother trying to use Thread.Abort() to recover.

A lightweight test harness will come in very handy when you're writing your real test runner GUI, command-line app, MSBuild task, Ant Task, ReSharper plugin, Visual Studio plugin, CruiseControl.Net plugin and on and on and on.  Each one of these can just fork off an instance when needed and consume the output and not worry about whatever all else is going on in the process that we need to protect the tests from...

(Oh, and whatever you do, don't assume your test harness will even be written in the same language as your test runner.  Don't reimplement Gallio.Runtime.Hosting!)

You don't need as much extensibility as you think...

MbUnit does an awful lot with .Net attributes.  I basically tried to make every part of the test execution lifecycle pluggable.

There are very few primitive concepts in MbUnit as a result.  So it's kind of cool that way.  But it's also rather hard to understand.  I'm sure there's an easier way to model this.

How many of you really grok how decorator pattern attributes work?  (If you want to understand this, go read Gallio.Framework.Pattern.PatternTest, PatternTestActions, PatternTestInstanceActions and IPattern.)

Dynamic tests are cool but static enumeration is important for tools

I'm rather fond of Gallio's concept of test steps.  They're the foundation of all of the dynamic test running capabilities in MbUnit.  They allow all sorts of nifty features such as representing each instance of a repeated test or data bound test separately and breaking big tests into smaller sections.

However, the structure of tests can't all be dynamic.  The test harness still needs to be able to find the high-level tests statically in some way so that a GUI can present them to the user.

MbUnit tries to strike a balance and offers many features for declaring tests statically and dynamically.  I am not aware of any other test framework that offers features resembling Gallio.Framework.TestContext, TestStep, TestLog and Tasks.  The data binding stuff is pretty cool too.  (Hint: This code is well worth studying... please borrow these ideas!)

NUnit is a bit more static.  As a result, it can make stronger assumptions about what tests look like and how they work.  That makes NUnit simpler and more efficient which is a good thing too.  (But I don't think it would hurt much for NUnit to adopt some more of MbUnit's ideas in this area.)

That said, when you find yourself defining your own reflection protocol maybe things have gone too far.  (But hey, the ReSharper integration is pretty cool.)

Get off Subversion!

Subversion is perfectly horrible for distributed software development.  It reserves the keys to the kingdom to a select group of "core committers" leaving everyone else out in the cold.

Use git!  And use GitHub or something like it that has built-in code review tools.

Jeff.

[Pat Kujawa]

Lots of great information, Jeff. Thanks for summarizing your thoughts for posterity.