How to force NUNIT3 to run test sequentially

[Christian Wittrock]

Hi,

We are using NUNIT3 for testing not only our code but also how it reacts to changes in HW IF availability (we have a device that can either be connected to via USB or BLE. When removing the USB connection a switch to BLE must happen automatically and vice versa). 

As NUNIT runs now we can see that it sometimes starts execution a second test when the first test is awaiting a switch in connection type. But as both tests makes changes to how the HW is connected the first test will fail in this case.

Because of this we need to restrict how NUNIT runs test. We need to make the tests runs in a sequential manner. Our tests are in all in a single assembly.

I have taken a look at some of the possible options:

--workers:    According to the documentation this ought to do the trick. But does this come with a performance penalty even when running a single test?

--inprocess: When using this option NUINT wont even load our assembly

--agents:     It does not seem clear what this option does. What is your definition of an agent? You already have thread limitation and assembly isolation. So, what does this option do?

We could solve the problem with the use of semaphores or similar. But if we can run with command line options we would prefer that.

Kind regards,

Christian

[Charlie Poole]

Hi Christian,

NUnit runs tests within an assembly sequentially by default - with an
exception. But first, some background...

--inprocess tells NUnit to run the tests within the console process.
It's equivalent to --process:Single. If you don't use it or --process,
the default is to create a separate agent process for each assembly.
We use the term "agent" exclusively to refer to process isolation.
Depending on what message you are seeing and the version of NUnit you
are running you may be encountering issue #1732 when using
--inprocess.

--agents tells NUnit how many agents may be active simultaneously. It
only applies if you use or default to --process=Multiple and it only
has effect if the number specified is less than the number of
assemblies. In that case, we will only load and run tests from that
number of agents at one time. People often set it to the number of
cores they have. However, it has no application with only one test
assembly being run.

--workers tells NUnit how many worker threads to create, in case you
have enabled parallel execution by use of attributes in your tests. If
you have not, then your tests are all run by the same worker, with the
exception noted below. The default number of workers is equal to the
number of processors. The overhead for setting up the queues and
workers is minimal but may be avoided by using --workers=0. When you
use that option, no queues and associated worker threads are created
and everything is run sequentially on the same thread.

The sole case where you could see some degree of parallelism, even
without enabling it in your code, is if you have some tests that
require the STA while others run (by default) in the MTA. In that
case, You may see some OneTimeSetUp code for different tests running
in paralllel, since different threads are used for code in the MTA and
STA. You can avoid this completely by use of --workers=0.

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Christian Wittrock]

Hi Charlie,

Thank you very much for the fast and detailed reply.

I am using version 3.2.0, so you are probably right that the version we have now has the #1732 bug. We will consider upgrading our NUNIT.

The tests did not seem to really run in parallel. It stopped the first test after having executed 1/4 of it, and it always stop the test at same place. But sometimes it does not stop the first test but completes it before continuing.. In the case where the execution of the first we put on hold, NUNIT went on to execute another test fully, after which it resumed the first test. So at no point did there seem to be more than a single thread running. There might have been multiple threads, but only one seemed to be active at a time.

We are running 100% default in relation to processes and parallelism. No test has been given a STA option (or MTA for that matter), so it is all down to how NUNIT runs. We will try to run with the --workers option to see if that works fine

/Christian

[Christian Wittrock]

Hi,

I have now upgraded to latest NUNIT release.

BTW, I have seen that you have written in another thread that the use of --inprocess and --x86 at the same time is not possible, But, when I use the --inprocess option I get the error message "Cannot run tests in process - a 32 bit process is required.". Not that I really need this, I was just curious. We are running on Windows 64 bit. Opening a console in 32bit mode doesn't work either.

We are are now running with --workers=0 but still see a problem with switching in the middle of a test execution

What we have been doing is:

  nunit3-console.exe ../Build/XXX.Test.dll --workers=0 --domain=Single --test=XXX.Test.PlatformTest.UsbPlatformTests,XXX.Test.PlatformTest.DevicePlatformTests,XXX.Test.PlatformTest.TransportSwitchingTests --result=../TestXXX.Test.xml;format=nunit2

Sometime while NUNIT was in the middle of running 'TransportSwitchingTests' it would jump to 'UsbPlatformTests' and execute those. Once done, it would resume the 'TransportSwitchingTests' test.

To overcome the problem we have been forced to run NUNIT in two steps:

  nunit3-console.exe ../Build/XXX.Test.dll --workers=0 --domain=Single --test=XXX.Test.PlatformTest.UsbPlatformTests,XXX.Test.PlatformTest.DevicePlatformTests --result=../TestXXX.Test.xml;format=nunit2

  nunit3-console.exe ../Build/XXX.Test.dll --workers=0 --domain=Single --test=XXX.Test.PlatformTest.TransportSwitchingTests --result=../TestTransportSwitching.Test.xml;format=nunit2

/Christian

[Rob Prouse]

Use workers=1. Zero means base it on the number of processors.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

[Christian Wittrock]

Update:

The spliting up of test and calling NUNIT multiple times instead of one, does not solve the problem. Even within a group of test one test may be put on hold and another one executed. So how to prevent this? 

[Christian Wittrock]

ok, I will try to set the workers to 1.

[Charlie Poole]

Hi Christian and Rob,

Actually, `--workers=1` doesn't cause the number of processors to be
used. Leaving the option off entirely is what causes that to happen.

The `--workers=1` option causes NUnit to set up 1 nominal worker
queue. I say nominal because there are actually several queues, with
only one active at a time. For example, there are separate queues for
tests allowed to run in parallel and tests not-allowed to run in
parallel - the default if you don't mark any tests as Parallelizable.
There are also separate queues for tests that run in the MTA and in
the STA. The result is that only one test runs at a time but the order
depends on how tests for various queues are nested. For example,
consider a fixture that contains one test that requires the STA. That
fixture's setup and most of its tests will run from one queue. Then
other tests will run from that queue until it is empty. Eventually, we
will switch to the STA queue and run the single STA test. Other STA
tests, from other fixtures, may run as well. When the STA queue is
exhausted, we will switch back to the MTA queue. Only then will the
fixture teardown for the original fixture run, interspersed with other
items on that queue.

The `workers=0` option bypasses all that, creating no queues at all
and running the tests in the same order as was used in NUnit V2. It
was originally included in 3.0 as a way to debug problems. If
something fails when using the new queue-based approach, it's handy to
be able to run it without queues in order to figure out where the
problem lies. However, a number of user comments have led me to
conclude that `--workers=0` is useful in itself. Maybe we should have
an easy-to-remember shortcut option for it, the way `--inprocess`
stands for `--process:Single`.

Charlie

[Christian Wittrock]

Hi,

We still have the problem with NUNIT not waiting for a test to complete, but puts it on hold and excutes another. This is a problem for us as we are using NUNIT in a sligthly different manner than what is normal for unit tests. We run our unit tests connected to real HW so that we not only get a module tested, but also how it responds to the current FW in our HW. Because of this we need NUNIT to run a test to end before starting a new one. It doesn't matter in which sequence the tests are executed, but once a test is commenced it must run to end without being interrupted by other tests.

How do we do this?

/Christian

[Charlie Poole]

What do you mean by "test" in this context? A test method? A fixture?

Are you using --workers=0?

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

[Christian Wittrock]

Hi Charlie,

By 'test' I do mean a test method. We have grouped the different tests, where each group is using its own fixture setup. But all groups are compiled into the same dll. I have seen a jump from one test to another within the same group but also to a test within a different group. I believe that after putting 'workers=0' we did no longer see jumps from one group to another. But I can't be 100% sure as the 'jumping' only happens at every odd run. 

One place where I have seen a 'jump' to a different test occuring, is where we have a Thread.Sleep within a polling routine. Please remember that we have sort of 'bent' the normal way of using unit tests as we use NUNIT to test module interaction with live HW (as well as using unit tests in the common way). Could it be that the NUNIT framework detects that the worker is at present not in use, as it is sleeping as so starts a new test, thinking that it can just go ahead?

We do find NUNIT a nifty tool with great value and we really appreciate your and your colleagues help.

[Charlie Poole]

NUnit has no code to detect your Sleep status and switch to another test. It does seem more likely that you would notice two tests running in parallel during a Sleep, however.

So let's check to see if any parallelism is involved. Do you use the Parallelizable attribute anywhere? Do you run multiple test assemblies?

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

[Christian Wittrock]

[Christian Wittrock]

Hi,

In no places do we have a STA or MTA attribute. We only use:

        [OneTimeSetUp]

        [OneTimeTearDown]

        [SetUp]

        [TearDown]

       [TestFixture]

       [Test]

We have 6 different classes, each contain several tests. Some of the classes overrides the setup functions in the base class, while others not. All use the same base class. 

Den fredag den 9. september 2016 kl. 16.11.07 UTC+2 skrev Charlie Poole:

[Charlie Poole]

What exactly do you do in your test, while waiting for hardware
availability? How is the test intended to be continued?

[Christian Wittrock]

One place where it happens often:

We have the class 'TransportSwithcingTests'. This class contains two tests 'TestNoDevice' and 'TestSwitch'. When these tests are executed the first test is always completed, but the second (and last) is sometimes only partially executed. It seems to always be at the same place the problem occurs and that is in between step 2 and step 3. The 'WriteLines' for step 2 are visible in the test log file, but not for step 3. To allow us to test switching from USB to BT and back we have our own USB hub, which allows us to enable/disable power on a specific port (XXXControl(true) means power on and a specific port).

This is part of the second test:

    [TestFixture]

    public class TransportSwitchingTests : PlatformTestFixture

    {

        [Test]

        public void TestSwitch()

        {

            // Test that switching between USB and Bluetooth works seamlessly

            try

            {

                Console.WriteLine("--------------------------------------------------------------------------------------------");

                Console.WriteLine("Step 1. Connections: USB and Dongle are both enabled.");

                Console.WriteLine("--------------------------------------------------------------------------------------------");

                DongleControl(true);

                WLPControl(true);

                Poll(() => device.GetTransportName().Equals("USB"), 10000);

                // Clear connection event queue

                connectionEvents.Clear();

                // Check that the basic methods work ok.

                int serial = device.ReadSerialNumber();

                // Simulate USB cable removed

                Console.WriteLine("--------------------------------------------------------------------------------------------");

                Console.WriteLine("Step 2. Connections: Disabling the USB connection.");

                Console.WriteLine("--------------------------------------------------------------------------------------------");

                WLPControl(false);

                Poll(() => device.GetTransportName().Equals("Bluetooth") && (device.ReadConnectionStatus().connected == true), 10000);

                Assert.AreEqual(serial, device.ReadSerialNumber());

                // Transport change should generate disconnected/connected events

                Assert.AreEqual(2, connectionEvents.Count);

                Assert.AreEqual(false, connectionEvents.Dequeue().connected);

                // Simulate USB cable inserted

                Console.WriteLine("--------------------------------------------------------------------------------------------");

                Console.WriteLine("Step 3. Connections: Enabling the USB connection.");

                Console.WriteLine("--------------------------------------------------------------------------------------------");

And this is the Poll function:

        public void Poll(Func<bool> predicate, int duration)

        {

            int elapsed = 0;

            while (elapsed < duration)

            {

                if (predicate())

                    return;

                else

                {

                    System.Threading.Thread.Sleep(200);

                    elapsed += 200;

                }

            }

            Assert.Fail("Condition was not met within specified duration.");

        }

[Christian Wittrock]

The interrupted test is completed in the end, once the interruption has completed

[Christian Wittrock]

Hi Charlie,

I have spent some time investigating what actually happens. And I can see that you are absolute right about NUNIT. It does only run single threaded. What happens is that a test fails, but instead of writing the failure message in the output, the next test is executed. And it is not until much later in the log any information will appear about the error.

My question is now, is it possible to get NUNIT to completely finalize the ongoing test before commencing a new test? I mean, so that all is logged and TearDown function called and so on.

[Christian Wittrock]

Hi Charlie,

Any ideas to why NUNIT is behaving as it is? 

[Charlie Poole]

Your last post puts it in a new light. Can you provide a reproducible
test case where this happens? If that's not feasible, please indicate
__exactly__ how your test is creating output that does not appear -
i.e. Console.Error.WriteLine, etc.

[Christian Wittrock]

Hi,

I figured that for the NUNIT output to indicate that a new test has been started, would either require running multiple threads or if the current test fails. So, I believe that the question is not a matter of threads but how the output is presented/interpreted. It may be that the output is as intended, and that I just need to know how to read it. I would have expected the log (when having only a single worker thread) to show:

Test Filters
    Test: WLP.Test.PlatformTest.TransportSwitchingTests
=> WLP.Test.PlatformTest.TransportSwitchingTests.TestNoDevice
--------------------------------------------------------------------------------------------
Disable dongle and USB
--------------------------------------------------------------------------------------------
=> WLP.Test.PlatformTest.TransportSwitchingTests.TestSwitch
--------------------------------------------------------------------------------------------
Step 1. Connections: USB and Dongle are both enabled.
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Step 2. Connections: Disabling the USB connection.
--------------------------------------------------------------------------------------------
Errors and Failures
1) Failed : WLP.Test.PlatformTest.TransportSwitchingTests.TestSwitch
  Condition was not met within specified duration.
  Expected: True
  But was:  False
=> WLP.Test.PlatformTest.UsbPlatformTests.TestBasic


But we are only seeing a summary of pass/fails:

Test Filters
    Test: WLP.Test.PlatformTest.TransportSwitchingTests
=> WLP.Test.PlatformTest.TransportSwitchingTests.TestNoDevice
--------------------------------------------------------------------------------------------
Disable dongle and USB
--------------------------------------------------------------------------------------------
=> WLP.Test.PlatformTest.TransportSwitchingTests.TestSwitch
--------------------------------------------------------------------------------------------
Step 1. Connections: USB and Dongle are both enabled.
--------------------------------------------------------------------------------------------
--------------------------------------------------------------------------------------------
Step 2. Connections: Disabling the USB connection.
--------------------------------------------------------------------------------------------
=> WLP.Test.PlatformTest.UsbPlatformTests.TestBasic
.

.

.

.

Errors and Failures
1) Failed : WLP.Test.PlatformTest.TransportSwitchingTests.TestSwitch
  Condition was not met within specified duration.
  Expected: True
  But was:  False


If the test fails immediately we can't even see the "=> WLP.Test.PlatformTest.TestXXX" that indicates the start of a test. When looking in the log it seems as if the test was not initiated at all (although not the case)

[Rob Prouse]

Christian,

The NUnit Console shows errors, failures and ignored tests at the end of the test run, not as they happen. For most users, intermingling the test results in with the test output would make it very hard to find the results amongst the noise.

Your requirements/expectations are different than what most users would want. If you want to output test status inline with the output, the simplest approach would be to wrap your tests in a try/catch/throw. A more versatile approach would be to create a custom attribute that wraps the tests for you. Something similar to the example ExpectedException attribute in the C# samples repository would do the trick.

Rob

On 16 September 2016 at 10:08, Christian Wittrock <christian...@gmail.com> wrote:

Hi,

[Charlie Poole]

The console output from a test run consists of three different parts:
1) Preliminary header info that NUnit produces.
2) Individual output created from your tests.
3) Summary information displayed by NUnit at the end of the run.

As Rob points out, the summary information is not created until the
entire test run completes. In your sample output, the summary starts
with the heading "Errors and Failures"

The individual test output coming directly from your tests is shown
preceded by a test label that tells you what test produced that
output. All the output from a test is saved and displayed only at the
end of that test. So the label diesb;t mark the start of a test, but
it's end.

If you want live output, displayed immediately rather than at the end
of the test, you should use Console.Error, TestContext.Error or
TestContext.Progress to display the value. The console will display
the first two in red, the third as normal output.

So, in summary, the info about a failed test will never appear
interspersed with the output from your tests and the label for the
test will not appear at all (using defaults) if the test never reaches
the point of creating output. If you do want to see labels for __all__
tests, you can use --labels:All to cause them to be produced.
Currently, those labels are still only displayed when the test ends,
but at least you know that it ran. We are working on a change to
--labels, which will give the option to display start labels as well
as pass/fail status.

Before going the route of a try/catch, which basically defeats NUnit's
normal mode of operation, I would suggest re-reading the documentation
carefully, particularly with regard to producing immediate output
through either an error wrote or use of TestContext.Progress. OTOH, if
understanding what's going on can resolve your problem, I hope this
explanation helps.