NUnit 3.0 Parallel Test Execution

[Charlie Poole]

Hi All,

I just pushed code to our Launchpad repository that implements
execution of tests in parallel. I'm posting for the benefit of those
who want to experiment with it (by building from source) and also to
let folks know what's coming and to give an opportunity for some
feedback on how we should develop this further, particularly in terms
of the limitations listed below.

This is not production-ready code, although it runs is quite stable
when used with truly independent unit tests. There are a few main
limitations:

1. Parallelism is between test fixtures. All the tests within a
fixture currently run on the same worker thread. This may actually be
a benefit for some cases, since it tends to minimize any accidental
interaction between tests.

2. More importantly, there is no way to limit parallelism once you
enable it. That is, if you specify multiple worker threads on the
command line, **all** fixtures are eligible to run in parallel. Our
next step is to figure out how to tell NUnit what can and want cannot
run in parallel and then to work out the implementation details. Ideas
are solicited.

3. The console runner is not yet able to activate this feature. Only
the direct-runner, which is used for testing the framework has the
-workers:n option available.

4. Only full NUnit has the feature. NUnitLite runs in a single thread as before.

[BTW, if you specify -workers:1 you will create a queue and one worker
thread. If you leave the option off, full NUnit will run the tests
just as it did before, without a separate queue or worker thread.]

Let me know if you try this and any ideas you have for improvement.

Charlie

[Tyrel Alastair Hunter]

Hello Charlie, how does this affect the repeat ability of random numbers?

—
Sent from Mailbox for iPhone

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at http://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/groups/opt_out.

[Charlie Poole]

Hi Tyrel,

It will have to be tested, of course, but I don't think it affects
anything. The tests are all loaded before execution starts and each
test case is assigned its own random seed. That seed is then used to
generate any random values used during the course of the test. So the
order of test execution should not matter.

This will change when we add the feature of generating test cases
dynamically and we'll have to figure out an alternative solution then.

Charlie

[Boris Abramovich]

Hello,

I was wondering, if i would like to refactor the framework so it could run Test Cases in parallel, not just Test Fixtures,

what would be the necessary code changes in your opinion ( in general ) ?

Thx,

Boris

[Charlie Poole]

Hi Boris,

I had to think about this for a bit, since it's actually a fairly
complex question. Anyway, here are my thoughts...

It's fairly trivial to cause test cases to be launched in parallel.
The class CompositeWorkItem has code for running the child tests in
any suite - i.e. fixture, setup fixture or assembly. If the child test
is below the fixture level, it executes it on the same thread where
the fixture setup was just run. Otherwise, it dispatches the test for
parallel execution. So, to run test cases in parallel, eliminate the
check and just dispatch everything.

However, this simple solution won't work except in specially contrived
cases that avoid all the pitfalls. In particular, if any of the test
cases (or the setup method) changes the state of the test class, the
tests will interfere with each other. Furthermore, if the test fixture
setup or any higher level setup is written to assume that all tests
run on the same thread, then tests run on a separate thread may break.

That's what led me to limit parallelism to the fixtures. The original
code for CompositeWorkItem ran the subordinate tests in parallel and
broke my own NUnit tests. I hypothesized that most fixtures - as
opposed to test cases - could probably run safely in parallel and this
turned out to be true, at least for my tests.

So, a more practical list of what's needed to run test cases in
parallel would include:

1. Implementing some ways to limit how much parallelism we have and
where. For example, we need to be able to designate tests that can't
be run in parallel with any other test, tests that can't run in
parallel with one another because they use a common resource, fixtures
whose test cases can or cannot be run in parallel, etc.

2. Allowing the user to designate a "level" at which parallelism takes
place. For example, this might be at the assembly level using parallel
processes, at the fixture level or a the test case level as you are
suggesting.

3. Not absolutely necessary, but I think desirable: allowing work
items to depend upon one another and splitting fixture and higher
level setup and teardown into two separate work items. All tests would
depend on the setup and the teardown would depend on all tests. Doing
this would allow us to schedule all subtests at one time rather than
waiting until each fixture setup completes.

4. Also desirable: having a setting or attribute that causes each test
case to run on a separate instance of the class. This would pretty
much eliminate the difficulty with tests that change the state of the
class and would provide much better separation of the tests.

5. Finally, implementing the dispatch mechanism as described at the
start of this note.

Maybe others, reading this, can come up with further necessary or
desirable steps.

Charlie