Nunit3-console: "Explicit" attribute and --where command-line switch - unexpected behaviour

[david.r...@viavisolutions.com]

Hello,

Windows 8.1

Visual Studio 2015

NUnit3-console from Nunit version 3.2, including the driver for Nunit 2.x

Test assembly built with Nunit 2.6.4 (via Nuget), any CPU, target framework v4.52.

Background:

We have a large suite of tests that we are preparing to migrate from Nunit 2.6.4 to Nunit 3.0.

The suite contains some test fixtures decorated with the 'Explicit' attribute.  Other fixtures have categories associated with them.  We exclude tests from categories such as 'Development' & 'Integration' during our build process.

In preparation for the migration, we are updating our console runners to version 3.2, with the aim of running our existing tests on them (via the 2.x driver) until all of the tests have been migrated.

Issue:

Contrary to our understanding of the documentation, the 'Explicit' attribute appears to be being ignored within the 3.2 console runner when a 'where' clause containing a 'not' condition is included on the command-line.

The documentation for 3.2 states:

Note that a not filter, which excludes certain tests, is not treated as an explicit selection and never causes an explicit test to be run.

We use this command-line with the 2.6.4 console runner to run each test fixture:

nunit-console <FixtureName>.dll /framework=4.0.30319 /exclude=Development,Integration,RabbitMQInstalled /stoponerror /noshadow /nologo /trace=Off

As expected, no tests or fixtures decorated with the 'Explicit' attribute are executed, neither are any tests that are categorised as 'Development', 'Integration' or 'RabbitMQInstalled'

When we execute the same test fixture with the test runner for Nunit 3.2, we use this command-line:

nunit3-console <FixtureName>.dll --framework=4.0.30319 --noheader --where "cat != Development && cat != Integration && cat != RabbitMQInstalled"  --stoponerror --trace=Off --noresult

As expected, tests that are categorised as 'Development', 'Integration' or 'RabbitMQInstalled' are not run.

As NOT expected, tests decorated with the 'Explicit' attribute ARE run.

We get the same result after the test fixture has been upgraded from Nunit 2.6.4 to Nunit 3.2.

Here is the code for a simple couple of test fixtures that demonstrate the issue:

using NUnit.Framework;

namespace ExplicitAttributeTests

{

    [TestFixture, Explicit("I should not run!")]

    public class TestFixtureWithExplicitAttribute

    {

        [Test]

        public void TestShouldNotRun()

        {

            Assert.Fail("I should not run!");

        }

    }

    [TestFixture]

    public class TestFixtureWithoutExplicitAttribute

    {

        [Test]

        public void TestShouldRun()

        {

        }

    }

}

Compile the code into a class library and execute it with a command-line similar to the one shown above.  For us, the console output is this:

C:\Users\raz59843\Documents\Visual Studio 2015\Projects\NunitExplicitTests\ExplicitAttributeTests\bin\Debug>"C:\Program Files (x8

6)\NUnit.org\nunit-console\nunit3-console" ExplicitAttributeTests.dll --framework=4.0.30319 --noheader --where "cat != Developmen

t && cat != Integration && cat != RabbitMQInstalled"  --stoponerror --trace=Off --noresult

Runtime Environment

   OS Version: Microsoft Windows NT 6.3.9600.0

  CLR Version: 4.0.30319.42000

Test Files

    ExplicitAttributeTests.dll

Test Filters

    Where: cat != Development && cat != Integration && cat != RabbitMQInstalled

Errors and Failures

1) Cancelled : C:\Users\raz59843\Documents\Visual Studio 2015\Projects\NunitExplicitTests\ExplicitAttributeTests\bin\Debug\ExplicitAttributeTests.dll

Cancelled by user

2) Cancelled : ExplicitAttributeTests

Cancelled by user

3) Failed : ExplicitAttributeTests.TestFixtureWithExplicitAttribute.TestShouldNotRun

I should not run!

at ExplicitAttributeTests.TestFixtureWithExplicitAttribute.TestShouldNotRun() in c:\users\raz59843\documents\visual studio 2015\P

rojects\NunitExplicitTests\ExplicitAttributeTests\TestFixtureWithExplicitAttribute.cs:line 11

4) Cancelled : ExplicitAttributeTests.TestFixtureWithoutExplicitAttribute

Test cancelled by user

Execution terminated after first error

Run Settings

    RuntimeFramework: 4.0.30319

    InternalTraceLevel: Off

    WorkDirectory: C:\Users\raz59843\Documents\Visual Studio 2015\Projects\NunitExplicitTests\ExplicitAttributeTests\bin\Debug

    StopOnError: True

    ImageRuntimeVersion: 4.0.30319

    ImageTargetFrameworkName: .NETFramework,Version=v4.5.2

    ImageRequiresX86: False

    ImageRequiresDefaultAppDomainAssemblyResolver: False

    NumberOfTestWorkers: 8

Test Run Summary

  Overall result: Failed

  Test Count: 1, Passed: 0, Failed: 1, Inconclusive: 0, Skipped: 0

    Failed Tests - Failures: 1, Errors: 0, Invalid: 0

  Start time: 2016-04-13 08:25:46Z

    End time: 2016-04-13 08:25:46Z

    Duration: 0.108 seconds

Is this a bug or are we misunderstanding something?

Regards

David Razzetti

[Charlie Poole]

Yes, this is issue #1141, which will be fixed in the next release. It
happens when the operands of an and operator are both negated. As a
workaround, you can do the standard conversion of !a && !b into !(a
|| b)

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[david.r...@viavisolutions.com]

Thanks for the bug confirmation.  I will try your suggestion of re-arranging the logic within the --where clause.