Running NUnit tests and examining results from within an NUnit3 Meta-Test

[Gerard Meszaros]

I have unit-test training exercises written in NUnit that I need to ensure still provide the same results. So I have "meta-tests" that run these tests and assert on the results obtained to ensure they match what I am expecting. 

I originally wrote these meta-tests in NUnit 2.2.8.  My meta-test project has references to each of the projects containing my exercises and I would do something like:

    TestDomain testDomain = new TestDomain();

    var project = NUnitProject.FromAssemblies(new string[] { "Exercise-03.dll});

    Test test = testDomain.Load(project);

to retrieve the tests and then to run them:

   TestResult result = testDomain.Run(NullListener.NULL);'   

I could then use ' ResultSummarizer' to get the stats for the test run to assert against. Or I could use a 'ResultVisitor' to inspect the individual test results.

This only works for running tests implemented in NUnit 2.2.8. I have some exercises where I use newer versions of NUnit (2.6.4) that won't work with this approach. And I would really like to upgrade all my exercises to the latest version of NUnit. So I wish to upgrade my meta-tests to use NUnit 3.5 so I can still regression test them.

What would be the best way to run NUnit 3.5 tests and assert the results programatically? I would prefer not to have to inspect an XML document but rather traverse an object tree (because it easier to figure out using Intellisense and code completion.)

Thanks,

Gerard

[Charlie Poole]

Hi Gerard,

NUnit does the same thing in it's own tests. If you look at the
TestBuilder class in the nunit.framework.tests assembly, as well as
some of the other classes in the same namespace, you'll see how we do
it. You can, of course, just grab the various utilities out of our
source on github.

This approach, however, means that you would not be running the tests
through the NUnit engine, but by direct use of the framework. Your
code would be bound to a particular framework version. The up side of
the approach is that you would be dealing with an object model rather
than the XML output.

If you want to use the (cleaner) approach of running tests through the
engine, you have to deal with the XML one way or another. If you look
at the nunit-gui project, you'll see that the Model folder contains
classes that wrap an XML element for a test or a test result. You
might get some use out of that.

If you ever come across the mountains to Seattle, let me know. We can
spend an afternoon hacking on this!

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Gerard Meszaros]

Hi Charlie,

Thanks for the quick reply. I'll have a poke about once I figure out where to find the various pieces (I'm having a bit of confusion about the various parts of NUnit 3.5. Do you have a one-pager somewhere describing how the various parts (Framework, Engine, GUI, Console, etc.) relate to each other, what they contain and when to use which one?)

No current plans to make it all the way over the mountains; I'm going as far as West Kelowna / Peachland over the holidays.

Thanks,

Gerard

[Charlie Poole]

Hi Gerard,

Such as it is...
https://github.com/nunit/docs/wiki/Architectural-Overview

Charlie

[Gerard Meszaros]

Looking at your code, I think all I need is the following to load and run a test assembly. Does this look right?

            _runner = new NUnitTestAssemblyRunner(new DefaultTestAssemblyBuilder());

            ITestAgency test = _runner.Load( Path.Combine(TestContext.CurrentContext.TestDirectory, "my-test-assembly.exe"), EMPTY_SETTINGS);

            var result = _runner.Run(TestListener.NULL, TestFilter.Empty);

            Assert.That(result.PassCount, Is.EqualTo(MockAssembly.Passed));

But I cannot figure out which "usings" I need to get this to compile. How do I get VS2015 to show me this in your solution? 

And I cannot get VS2015 to find namespaces:

using NUnit.Tests;

using NUnit.Tests.Assemblies;

using NUnit.Framework.Interfaces;

using NUnit.Framework.Internal;

As a non-native VS & nuget user, I am having trouble managing references. I installed NUnit 2.2.8 and 2.6.4 using the MSI installers. Only NUnit 2.6.4 is showing up in Add References. When I use 

Which nuget package do I need to install to see these?  Nuget Package Manager brings up a huge number of packages when you type `NUnit` into the search box. I have installed NUnit, NUnit.Console, NUnit.ConsoleRunner, NUnit.Engine, NUnit.Extenstion.NUnitProjectLoader, NUnit.Extension.NUnitV2Driver/ResultWriter, NUnit.Extesion.VSProjectLoader and NUnit3TestAdapter. But only nunit-agent(-x86) and nunit.engine(.api) show up in references. And packages.config only contains:

<?xml version="1.0" encoding="utf-8"?>

<packages>

  <package id="NUnit" version="3.5.0" targetFramework="net45" />

  <package id="NUnit.ConsoleRunner" version="3.5.0" targetFramework="net45" />

  <package id="NUnit3TestAdapter" version="3.6.0" targetFramework="net45" />

</packages>

What am I doing wrong?

Thanks,

Gerard

BTW, The diagram "images/nunit-xtp-2008.png" link just takes me back to the wiki root. (https://github.com/nunit/docs/wiki)

[Charlie Poole]

Hi Gerard,

First, you need to decide whether you want to use a low-level or
high-level solution. The code snippet you gave is low-level, using
only the nunit framework. If you are using that approach, you can't
run nunit v2 tests at all. You can only run NUnit 3.5 tests because
you will have a reference to the nunit 3.5 framework. (That's why it's
"low") The framework source is in the GitHub nunit/nunit repo and is
installed using the nuget NUnit package.

All the other packages you mention are (different) high-level ways to
run tests. NUnit.ConsoleRunner is the console runner and engine.
NUnit.Engine is just the engine, for folks who are writing high-level
runners. The extensions are ... well... extensions. The test adapter
is an adapter for running NUnit tests under the Visual Studio Test
Explorer.

Installing the NUnit package alone should give you visibility to all
the public members of the framework assembly, provided you have using
statements for the classes you need. In particular, the
NUnitTestAssemblyRunner and DefaultTestAssemblyBuilder are in
NUnit.Framework.Api. So long as you are using that approach, get rid
of all the other references.

Good luck and keep asking questions!

Charlie

[Gerard Meszaros]

OK, this is going to seem like a really stupid qquestion. Why can't I add a reference to nunit.framework-3.5 to my project?  In your Solution, it is under Project->Solution. In mine, of course it isn't. Because I'm in a different solution. I can only find one for nunit.framework 2.6.4.143 when I do Add Reference->Assemblies->Extensions. Nuget Manager tells me that 3.5 is installed in my project but 

On Thursday, December 15, 2016 at 1:54:43 PM UTC-7, charlie wrote:

Hi Gerard,

First, you need to decide whether you want to use a low-level or
high-level solution. The code snippet you gave is low-level, using
only the nunit framework.  If you are using that approach, you can't 

run nunit v2 tests at all.

So then which project/package should I be looking for for examples? 

 

You can only run NUnit 3.5 tests because
you will have a reference to the nunit 3.5 framework. (That's why it's
"low") The framework source is in the GitHub nunit/nunit repo and is
installed using the nuget NUnit package.

 

Not sure I understand the significance of "having a reference to the nunit 3.5 framework". Doesn't each project have it's own set of references? Which could be different versions of the framework. 

All the other packages you mention are  (different) high-level ways to
run tests. NUnit.ConsoleRunner is the console runner and engine.
NUnit.Engine is just the engine, for folks who are writing high-level
runners. The extensions are ... well... extensions. The test adapter
is an adapter for running NUnit tests under the Visual Studio Test
Explorer.

Installing the NUnit package

 

How do I do this? The instructions say to use NuGet but I'm not having any luck with that. The alternative is a zip file and that hardly seems like "installation". How do I get it to show up in the list of references I can add? And which category should it show up in? Or is having it in packages.config enough. (Is your solution a good example of how to do this? It seems like everything is within the solution while I'm trying to bring things in from my NUnit installation.)

alone should give you visibility to all
the public members of the framework assembly, provided you have using
statements for the classes you need. In particular, the
NUnitTestAssemblyRunner and DefaultTestAssemblyBuilder are in
NUnit.Framework.Api. So long as you are using that approach, get rid
of all the other references.

Your TestAssemblyRunnerTests extends ITestListener.  VS2015 says ITestListener is in NUnit.Framework.Interfaces which cannot be found in my solution.

Good luck and keep asking questions!

What's with the NET-2.0, NET-3.5, etc.? Should I care which one I'm looking in? It looks like the same files exist under each one; are they the same files or just named the same?

Thanks,

Gerard

p.s. This "plumbing" is always the worst part of learning to use a development environment. I need a plumber to help me get it set up. ;-)

[Charlie Poole]

Hi Gerard,

I'm a little confused about whether we are talking about the test or
the meta-test project here.

Can you give me a small example of what you want to do? Pseudo-code would be OK.

When you install the NUnit nuget package as a reference in your
project, everything is done for you. You don't care about the
structure of the package because the files you need (probably just the
framework dll) is copied to your output directory from the particular
subdirectory that applies, depending on what framework you are
targeting. So, if you are targeting .NET 4.5, the framework will be
copied from the net45 directory. If you are looking at the source
code, yes, they are all the same files.

Give me a little sample and I'll try to fill it in for you.

Charlie

[Gerard Meszaros]

On Thursday, December 15, 2016 at 4:51:05 PM UTC-7, charlie wrote:

Hi Gerard,

I'm a little confused about whether we are talking about the test or
the meta-test project here.

 

MetaTest project. The Test project contains a single test with a single statement Assert.fail("Not implemented yet!"); It references NUnit.Framework-2.2.8. 

Can you give me a small example of what you want to do? Pseudo-code would be OK.

My MetaTests all look something like this:

        [Test]

public void TestExercise01()

{

    TestResult result = GetTestResultsForDLL("Exercise-01-Test.dll");  

    verifyResultCounts(result, 1, 0, 1);

    assertTestCaseFailedWithMessagePart("Exercise1TestCase", "Test1", "Not Implemented Yet!");

}

 

When you install the NUnit nuget package as a reference in your
project, everything is done for you. You don't care about the
structure of the package because the files you need (probably just the
framework dll) is copied to your output directory from the particular
subdirectory that applies, depending on what framework you are
targeting. So, if you are targeting .NET 4.5, the framework will be
copied from the net45 directory. If you are looking at the source
code, yes, they are all the same files.

I'm having trouble getting the MetaTest project to see the various namespaces in which are declared the classes that I am trying to use from your internal test project. (I copied the following code from your TestAssemblyRunnerTests class and I cannot get it to compile because it cannot see many of your classses. )  

            _runner = new NUnitTestAssemblyRunner(new DefaultTestAssemblyBuilder());

            ITestAgency test = _runner.Load( Path.Combine(TestContext.CurrentContext.TestDirectory, "my-test-assembly.exe"), EMPTY_SETTINGS);

            var result = _runner.Run(TestListener.NULL, TestFilter.Empty);

            Assert.That(result.PassCount, Is.EqualTo(MockAssembly.Passed));

In NUnit 2.2.8 I did the following:

Thanks,

Gerard