RE: Running NUnit tests in parallel

[Charlie Poole]

Hi Alan,

 

I cc'ed the list - but maybe not the one you expected. :-) NUnit-users is no longer used and

SourceForge sends a confusing notice to you when you try to sign up. You'll eventually

get a "rejection" notice telling you about the new list.

 

In NUnit 2.5, parallel tests are supported through pNUnit but are not completely

integrated with normal NUnit tests as you suggest. Yet. However, it's coming.

 

Charlie

---

From: Alan McGovern [mailto:alan.m...@gmail.com]
Sent: Monday, December 22, 2008 4:27 PM
To: Charlie Poole
Subject: Running NUnit tests in parallel

Hey,

I believe my request to join the nunit mailing list is pending, so please CC the list when you reply to this in case the info comes in useful to someone else.

I was just looking at upgrading the NUnit version I depend on as parametrised tests are one thing that I'll find pretty useful. Another thing that would drastically cut down on the runtime for my tests would be the ability to run multiple tests in parallel. I took a quick look, but NUnit doesn't appear to support this. What I was thinking of would be that I would mark a test with a [Parallel] attribute, then NUnit would execute that test in threadpool thread and continue executing the rest of the tests in the current fixture/next fixture.

Is something like that possible?

Alan.

[Charlie Poole]

Hi Alan,

Replying on list since your suggestion is sure to have some
general interest.

> > In NUnit 2.5, parallel tests are supported through pNUnit
> but are not
> > completely integrated with normal NUnit tests as you suggest. Yet.
> > However, it's coming.
>

> I took a quick look at pNUnit, but it didn't really appear to
> cover my usecase. It seemed to be greared at *distributed*
> NUnit test running.

Yes, pnunit is geared toward distributed testing with several
tests running in parallel and talking to one another.

> What I want is to just complete my tests in parallel so the
> whole suite completes in 10 seconds instead of 20. The basic
> idea is that for all the long-running TestFixtures, i drop a
> [Parallel] attribute onto the fixture. Then NUnit would run
> all testfixtures with the [Parallel] attribute first, and in
> parallel (maybe defaulting to 2 simultaneous parallel
> testfixtures at a time). This way while all my 'fast' tests
> are running sequentially, my blocking/slow tests can execute
> in parallel.
>
> Is this covered by pNUnit?

No, it's not what pNUnit is for.

> If not, would there be interest in
> me trying to hack in support for that?

Sure, with some reservations - see below.

> This is the most basic
> form of parallelism, and should be the easiest to implement.

I agree. The reason we don't have it is twofold:
1) We didn't have tests on threads until the beta of 2.5
2) The pNUnit app came to us ready to go from Codice software.

> Test level parallelism would be a little trickier because
> running [Setup] and [TearDown] becomes a little tricky. That
> might be fixable by slapping in a few [ThreadStatic]
> attributes on member variables. However, I'd be quite happy
> with fixture level parallelism.

I think parallelism between tests in the same fixture would
probably work best if we created a separate instance for
each test.

I'd be happy to see you work on this with a few provisos:
1) You need to work with CVS head, in order for it to be
useful.
2) There's a chance it won't make 2.5, since I don't want
to delay it for yet another feature - but if you follow
point #1, it will be ready for injection in a 2.6 release.
3) I think you need to discuss on list just how the attribute
would work. As you explained it, I understand that [Parallel]
on a fixture would mean "in parallel with other fixtures that
are so marked." It could also be taken to mean "Run all the
test cases in this fixture in parallel" so we should resolve
both how it should work and the best way to express it for
all NUnit users. We may want to look at how other frameworks
express this.

BTW, if you look at the code, you'll see that it IS currently
possible to run assemblies in parallel by changing a define
and rebuilding. However, I think parallel test assemblies is
best handled by using multiple processes rather than just
separate threads.

I hope I haven't discouraged you and that this can start a
bit of a dialog on the list about how it should work.

Charlie

> Alan.
>
> > Charlie
> >
> >   _____

[Alan McGovern]

Hey,

> Test level parallelism would be a little trickier because
> running [Setup] and [TearDown] becomes a little tricky. That
> might be fixable by slapping in a few [ThreadStatic]
> attributes on member variables. However, I'd be quite happy
> with fixture level parallelism.

I think parallelism between tests in the same fixture would
probably work best if we created a separate instance for
each test.

A TestFixture with [ThreadStatic] on every member variable would have essentially the same affect as instantiating a new object for every test, except it's even easier to support. So we're on the same page here :)

I'd be happy to see you work on this with a few provisos:
1) You need to work with CVS head, in order for it to be
useful.

I figured as much, that's a pretty standard requirement.
 

2) There's a chance it won't make 2.5, since I don't want
to delay it for yet another feature - but if you follow
point #1, it will be ready for injection in a 2.6 release.

It'd probably take  me a few days to get familiar with the codebase. Once you take into account the christmas holidays and all that entails, it's unlikely I'll have a working patch until mid january. Though I will keep the list posted on any progress I make.

3) I think you need to discuss on list just how the attribute
would work. As you explained it, I understand that [Parallel]
on a fixture would mean "in parallel with other fixtures that
are so marked." It could also be taken to mean "Run all the
test cases in this fixture in parallel" so we should resolve
both how it should work and the best way to express it for
all NUnit users. We may want to look at how other frameworks
express this.

 

The way I'm looking at it is that the main (and only?) reason for running tests in parallel is so that they complete faster. Other than that, there is no benefit to being able to run tests in parallel. So, with that in mind, the main aim of the [Parallel] attribute is to easily parallelise tests. So what i'm proposing is:

ParallelFixtureAttribute : TestFixtureAttribute

You just drop that attribute on top of any testfixture which you want to run in parallel. The NUnit runner would (by default) grab all the assemblies with this attribute and start executing them in parallel while the main thread deals with all the regular fixtures marked with [TestFixture].

If you want more complex handling, you could have a:

ParallelAttribute : TestAttribute

In this case, what would happen is that if a [TestFixture], or [ParallelFixture] has a test marked with [Parallel], a new testfixture object will be created and this test would be executed in parallel to the regular tests. This way you don't run into issues with shared member variables in the test fixture etc. Though I do think that Fixture level parallelism would more than likely be all you'd need.

So does that sound? Is my assumption that the only reason for parallel tests is to speed up execution a good enough one, or are there other use cases I haven't thought of?

Alan.
 

[Charlie Poole]

Hi Alan,

 

> Test level parallelism would be a little trickier because
> running [Setup] and [TearDown] becomes a little tricky. That
> might be fixable by slapping in a few [ThreadStatic]
> attributes on member variables. However, I'd be quite happy
> with fixture level parallelism.

I think parallelism between tests in the same fixture would
probably work best if we created a separate instance for
each test.

A TestFixture with [ThreadStatic] on every member variable would have essentially the same affect as instantiating a new object for every test, except it's even easier to support. So we're on the same page here :) 
 

True... it just puts more burden on the user, but maybe that's where it belongs. A greater danger,

in fact, is with static members, if they are set in SetUp rather than TestFixtureSetUp. We'd have

to publish guidelines for use.

 

But it does seem that running fixtures in parallel is a whole lot easier!

 

I'd be happy to see you work on this with a few provisos:
1) You need to work with CVS head, in order for it to be
useful.

I figured as much, that's a pretty standard requirement.
 

2) There's a chance it won't make 2.5, since I don't want
to delay it for yet another feature - but if you follow
point #1, it will be ready for injection in a 2.6 release.

It'd probably take  me a few days to get familiar with the codebase. Once you take into account the christmas holidays and all that entails, it's unlikely I'll have a working patch until mid january. Though I will keep the list posted on any progress I make.

3) I think you need to discuss on list just how the attribute
would work. As you explained it, I understand that [Parallel]
on a fixture would mean "in parallel with other fixtures that
are so marked." It could also be taken to mean "Run all the
test cases in this fixture in parallel" so we should resolve
both how it should work and the best way to express it for
all NUnit users. We may want to look at how other frameworks
express this.

 

The way I'm looking at it is that the main (and only?) reason for running tests in parallel is so that they complete faster. Other than that, there is no benefit to being able to run tests in parallel. So, with that in mind, the main aim of the [Parallel] attribute is to easily parallelise tests. So what i'm proposing is:

ParallelFixtureAttribute : TestFixtureAttribute

You just drop that attribute on top of any testfixture which you want to run in parallel. The NUnit runner would (by default) grab all the assemblies with this attribute and start executing them in parallel while the main thread deals with all the regular fixtures marked with [TestFixture]. 

 

Alternatively, you could have ParallelAttribute : PropertyAttribute. That way it could be used

in combination with TestFixtureAttribute without duplicating all its constructors (2.5 has generic

and parameteried test fixtures) and it would be able to work with future attributes that may

apply to classes containing tests. Generally, NUnit has moved toward orthogonality with

a single attribute serving a single function - compare Description, Combinatorial, etc.

 

By inheriting from PropertyAttribute, you could set a property (PARALLEL, most likely)

on the test and NUnit would not have to do any furthe reflecting beyond what it already

does. TestSuite (which does the running) would have to look for the property and act

accordingly.

 

The one thing that's not clear is how to force the parallel tests to run first - or frankly,

whether to do that. It seems like ordering is orthogonal to parallel execution. In your

particular use case you want those tests to run first, but not every use case would

necessarily want that.

 

If you want more complex handling, you could have a:

ParallelAttribute : TestAttribute

In this case, what would happen is that if a [TestFixture], or [ParallelFixture] has a test marked with [Parallel], a new testfixture object will be created and this test would be executed in parallel to the regular tests. This way you don't run into issues with shared member variables in the test fixture etc. Though I do think that Fixture level parallelism would more than likely be all you'd need.
 

I agree.

 
So does that sound? Is my assumption that the only reason for parallel tests is to speed up execution a good enough one, or are there other use cases I haven't thought of?
 

Sounds like a reasonable start, but what do I know? Anyone else?

 

Charlie 
Alan.
 

[Alan McGovern]

Hey,

On Wed, Dec 24, 2008 at 4:43 AM, Charlie Poole <cha...@nunit.com> wrote:

Hi Alan,

 

> Test level parallelism would be a little trickier because
> running [Setup] and [TearDown] becomes a little tricky. That
> might be fixable by slapping in a few [ThreadStatic]
> attributes on member variables. However, I'd be quite happy
> with fixture level parallelism.

I think parallelism between tests in the same fixture would
probably work best if we created a separate instance for
each test.

A TestFixture with [ThreadStatic] on every member variable would have essentially the same affect as instantiating a new object for every test, except it's even easier to support. So we're on the same page here :) 
 

True... it just puts more burden on the user, but maybe that's where it belongs. A greater danger,

in fact, is with static members, if they are set in SetUp rather than TestFixtureSetUp. We'd have

to publish guidelines for use.

 

But it does seem that running fixtures in parallel is a whole lot easier!

Yeah, static members are going to be problematic in you're running individual tests in parallel, whether they're in FixtureSetup or SetUp. If we go with the instantiate a new TextFixture object to run the tests in parallel, then we'd have to run the FixtureSetup method again, so you're always gonna have a problem.

Sounds good.

 

The one thing that's not clear is how to force the parallel tests to run first - or frankly,

whether to do that. It seems like ordering is orthogonal to parallel execution. In your

particular use case you want those tests to run first, but not every use case would

necessarily want that.

 The main reason driving this is that if you mark a Fixture as [Parallel], then it's probably a lengthy set of tests, so you want to start executing these as soon as possible and not leaving them til the end. For example, doing a lot of network IO or disk IO. One particular set of tests I have take ~200ms per test simply due to socket setup delays.

If you don't grab all the [Parallel] tests at the start, then what can happen is that all your lengthy tests will execute towards the end of your fixture, when there are no sequential tests left and so you won't get the full benefit of parallel execution.

Actually, this brings me to a revelation. Why are we bothering with attributes for fixture level stuff. Would a setting in a .config file not be better? Just add a "ParallelFixtures<bool>" attribute. Why should only certain tests be cherry picked to be parallel. This would only really be important for individual tests, not fixtures. Fixtures are typically 100% parallelisable so in the common use case a user would have to switch every [TestFixture] to [ParallelFixture] (or whatever) manually. So, would it make more sense to have this as a global option to enable/disable it for all TestFixtures.

Alan.

[Mike Chess]

Using a configuration file technique seems to be a bad idea to me.  For example:  I may be a developer that is having an issue working with parallel tests and I want another developer to take a look at the problem.  So I check my tests into version control so that they can have access to them.  They then run the tests and report no problems.  In this scenario I have the parallel option checked in my NUnit config while they do not.

 

If parallel is expressed as an attribute of the fixture or the test method my intent for the tests are clearly expressed to other developers who may be called on to look at my tests.  A config option for the application running the tests seems to be unclear.

 

Mike

 

<snip>

Actually, this brings me to a revelation. Why are we bothering with attributes for fixture level stuff. Would a setting in a .config file not be better? Just add a "ParallelFixtures<bool>" attribute. Why should only certain tests be cherry picked to be parallel. This would only really be important for individual tests, not fixtures. Fixtures are typically 100% parallelisable so in the common use case a user would have to switch every [TestFixture] to [ParallelFixture] (or whatever) manually. So, would it make more sense to have this as a global option to enable/disable it for all TestFixtures.

Alan.

 

<snip>

[Charlie Poole]

Hi Mike,

 

I see your point, but I don't think we can get around it very easily.

 

1. I don't think Parallel should be more than a hint to NUnit. It may run all the

Parallel fixtures in parallel, some number of them or none of them. That's different

from a test that can ONLY run in parallel - a distributed scenario.

 

2. If the developer doesn't KNOW that his test only fails when run in parallel

with some othr test, he really isn't doing his job. Parallel stuff is always going

to be intrinsically more difficult.

 

That said, I think I agree with you that we can't have just a config setting. We

may have some settings that generally enable or disable parallel feature but

I think we need to identify those fixtures that are capable of running in parallel.

 

BTW, I'm taking "config" to mean "some kind of setting" - not the actual

config file. NUnit is slowly using it's config settings in favor of defaults in

the Settings.xml file and project-specific settings in the .nunit file.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Mike Chess
Sent: Wednesday, December 24, 2008 6:49 AM
To: nunit-...@googlegroups.com
Subject: [nunit-discuss] Re: Running NUnit tests in parallel

[Charlie Poole]

Hi Alan,

Yeah, static members are going to be problematic in you're running individual tests in parallel, whether they're in FixtureSetup or SetUp. If we go with the instantiate a new TextFixture object to run the tests in parallel, then we'd have to run the FixtureSetup method again, so you're always gonna have a problem. 

 

That's why James called TestFixtureSetup as implemented a mistake. He thinks it should have

been allowed only on a static method. That's water under the bridge, however. I guess we just

need to warn the users about any sharp knives we leave laying around. :-) 

 The main reason driving this is that if you mark a Fixture as [Parallel], then it's probably a lengthy set of tests, so you want to start executing these as soon as possible and not leaving them til the end. For example, doing a lot of network IO or disk IO. One particular set of tests I have take ~200ms per test simply due to socket setup delays.

If you don't grab all the [Parallel] tests at the start, then what can happen is that all your lengthy tests will execute towards the end of your fixture, when there are no sequential tests left and so you won't get the full benefit of parallel execution.
 

I understand that's your motivation, but I also have a lot of experience with people finding

their own uses - uses I never thought of - for NUnit features. So I hesitate to couple the

two things. BTW, this is only a workaround, but the one place that NUnit currently guarantees

order is that it runs assemblies in the order they are listed in an nunit file.

 
Actually, this brings me to a revelation. Why are we bothering with attributes for fixture level stuff. Would a setting in a .config file not be better? Just add a "ParallelFixtures<bool>" attribute. Why should only certain tests be cherry picked to be parallel. This would only really be important for individual tests, not fixtures. Fixtures are typically 100% parallelisable so in the common use case a user would have to switch every [TestFixture] to [ParallelFixture] (or whatever) manually. So, would it make more sense to have this as a global option to enable/disable it for all TestFixtures.
 

I'm taking [Parallel] to mean "capable of running in parallel" It's possible - even easy to

write fixtures that will break when run in parallel. It only takes some bad code. :-)

 

So there has to be some way to designate fixtures as parallelizable.

 

To make it easier, we could setting at the assembly level - in fact that's

one reason to prefer [Parallel] over [ParallelFixture] - if you had

[assembly:Parallel] then all fixtures could be treated as parallelizable.

 

A global setting to disable it would also be a possibility, if needed

for debugging, for example.

 

 

Great discussion - let me know how I can help you get started with

this. Have you built the head yet?

[Alan McGovern]

Hey,

Some comments inline and then some preliminary results after :)

On Wed, Dec 24, 2008 at 7:58 PM, Charlie Poole <cha...@nunit.com> wrote:

Hi Alan,

Yeah, static members are going to be problematic in you're running individual tests in parallel, whether they're in FixtureSetup or SetUp. If we go with the instantiate a new TextFixture object to run the tests in parallel, then we'd have to run the FixtureSetup method again, so you're always gonna have a problem. 

 

That's why James called TestFixtureSetup as implemented a mistake. He thinks it should have

been allowed only on a static method. That's water under the bridge, however. I guess we just

need to warn the users about any sharp knives we leave laying around. :-) 

I think most developers know the pitfalls of static members and so can be trusted enough to do the right thing. If they don't, well, they'll learn once things start breaking on them ;)

 The main reason driving this is that if you mark a Fixture as [Parallel], then it's probably a lengthy set of tests, so you want to start executing these as soon as possible and not leaving them til the end. For example, doing a lot of network IO or disk IO. One particular set of tests I have take ~200ms per test simply due to socket setup delays.

If you don't grab all the [Parallel] tests at the start, then what can happen is that all your lengthy tests will execute towards the end of your fixture, when there are no sequential tests left and so you won't get the full benefit of parallel execution.
 

I understand that's your motivation, but I also have a lot of experience with people finding

their own uses - uses I never thought of - for NUnit features. So I hesitate to couple the

two things. BTW, this is only a workaround, but the one place that NUnit currently guarantees

order is that it runs assemblies in the order they are listed in an nunit file.

Well, lets take the idea of parallel tests a little further: How can you guarantee order if you're running tests in parallel? If there are two test threads executing fixtures A and B. The first thread could complete and then execute C, D, E and F before FixtureB has completed. So whether or not we explicitly change the ordering, parallel tests will implicitly change the ordering anyway.

Actually, this brings me to a revelation. Why are we bothering with attributes for fixture level stuff. Would a setting in a .config file not be better? Just add a "ParallelFixtures<bool>" attribute. Why should only certain tests be cherry picked to be parallel. This would only really be important for individual tests, not fixtures. Fixtures are typically 100% parallelisable so in the common use case a user would have to switch every [TestFixture] to [ParallelFixture] (or whatever) manually. So, would it make more sense to have this as a global option to enable/disable it for all TestFixtures.
 

I'm taking [Parallel] to mean "capable of running in parallel" It's possible - even easy to

write fixtures that will break when run in parallel. It only takes some bad code. :-)

True. The trivial case is that FixtureA and FixtureB both try to use a socket at port 1000. Sequentially this is fine. In parallel, not so good.

 

So there has to be some way to designate fixtures as parallelizable.

 

To make it easier, we could setting at the assembly level - in fact that's

one reason to prefer [Parallel] over [ParallelFixture] - if you had

[assembly:Parallel] then all fixtures could be treated as parallelizable.

Yes, I actually much prefer this idea than a setting hidden in some config file. I didn't think of it myself, good call!

 

A global setting to disable it would also be a possibility, if needed

for debugging, for example.

Yeah, it'd be great if GUI's were able to control this, this was my motivation for having a setting. Basically the assembly:Parallell attribute could control whether or not [Parallel] tests will actually be run in parallel by default and GUI's could override that should they wish to.

 

 

Great discussion - let me know how I can help you get started with

this. Have you built the head yet?

 

Yeah, I checked out the code earlier and did a test build. However, before that I did a little benchmarking. I wanted to see what the real-world benefit was before I wasted time trying to implement something that gave a negligible boost. The source code for the sample parallel runner is attached to the email. Basically I execute all the NUnit tests in the current assembly in parallel and see how long it takes. If you want to test this code on your own NUnit tests, just compile Main.cs with all your NUnit tests and then execute the resulting console application.

So here's some results from running the 220 odd tests for MonoTorrent:

1 Thread:  16.8 seconds
2 Theads: 11.6 seconds
3 Threads: 10.2 seconds
4 Threads: 9.0 seconds
5 Threads: 9.1 seconds

So yes, there is most definitely a benefit to having parallel tests. The other thing is that all my tests worked fine without modification :)

Now, for my critiques on NUnit itself. Note, this is the result of a brief look at the code starting with the nunit-gui.exe projoect file, so my observations could be way off ;)

When an assembly is loaded, it is parsed into a series of 'Test' instances automagically. When you hit 'F5' to execute the tests, you end up in TestSuite.Run. This is great from a design point of view because you hit test.Run() and get the result set when the call returns, but this approach is pretty damn hard to multithread. There is no clear division where you can say "This is where i can put my threads".

If you want parallel threads, the whole TestSuite.Run method would have to be reworked. What you really want to have is something similar to my testcase where you have a list of TestFixtures which contain a list of Tests which can be executed in parallel.
What you have at the moment is a load of tests containing other tests, which may or may not contain more tests and you just keep recursing until you finally hit an actual Test and execute it. With this approach, where do i spin off the threads?

I did a quick hack of the code to test what would happen if I added a check to see if all the Tests in the current TestSuite are 'NUnitTestFixtures" and then executing these in parallel. Psuedo code for TestSuite.RunAllTests:

if (tests.TrueForAll(delegate(Test t) { return t is NUnitTestFixture; })) {
    foreach (Test test in Tests)
        ThreadPool.QueueUserWorkItem (delegate { ProcessTestNormally (test); });
    WaitForAsyncTestsToComplete ();
}
else {
    foreach (Test test in Tests)
        ProcessNormally (test);
}

However this revealed that the design of how the tests are executing cannot support running tests in parallel. I kept getting errors in the NUnit code caused by the multiple worker threads stepping on each other. So, rather than wasting *too* much time scouting around the code trying to figure out how best to work this out, do you have any suggestions? I'm all ears :)

Alan.

p.s. I hope that wasn't too confusing ;)

[Charlie Poole]

Hi Alan,

 

Well, lets take the idea of parallel tests a little further: How can you guarantee order if you're running tests in parallel? If there are two test threads executing fixtures A and B. The first thread could complete and then execute C, D, E and F before FixtureB has completed. So whether or not we explicitly change the ordering, parallel tests will implicitly change the ordering anyway.
 

NUnit doesn't currently guarantee order. I thought you were proposing we guarantee a certain order. 
 

Of course, if we added ordering - for example with an attribute - we would have to figure out what it meant

when we also have parallel tests.

 

Great discussion - let me know how I can help you get started with

this. Have you built the head yet?

 

Yeah, I checked out the code earlier and did a test build. However, before that I did a little benchmarking. I wanted to see what the real-world benefit was before I wasted time trying to implement something that gave a negligible boost. The source code for the sample parallel runner is attached to the email. Basically I execute all the NUnit tests in the current assembly in parallel and see how long it takes. If you want to test this code on your own NUnit tests, just compile Main.cs with all your NUnit tests and then execute the resulting console application.

So here's some results from running the 220 odd tests for MonoTorrent:

1 Thread:  16.8 seconds
2 Theads: 11.6 seconds
3 Threads: 10.2 seconds
4 Threads: 9.0 seconds
5 Threads: 9.1 seconds 

 

I saw substantially better improvement when parallelizing at the assembly level - probably due

to lower overhead. 

So yes, there is most definitely a benefit to having parallel tests. The other thing is that all my tests worked fine without modification :)

Now, for my critiques on NUnit itself. Note, this is the result of a brief look at the code starting with the nunit-gui.exe projoect file, so my observations could be way off ;)

When an assembly is loaded, it is parsed into a series of 'Test' instances automagically. When you hit 'F5' to execute the tests, you end up in TestSuite.Run. This is great from a design point of view because you hit test.Run() and get the result set when the call returns, but this approach is pretty damn hard to multithread. There is no clear division where you can say "This is where i can put my threads".
 

Yup. That's one reason why this whole design needs to change in NUnit 3.0. However, I think it may be more doable than it seems at first glance.

 
If you want parallel threads, the whole TestSuite.Run method would have to be reworked. What you really want to have is something similar to my testcase where you have a list of TestFixtures which contain a list of Tests which can be executed in parallel.
What you have at the moment is a load of tests containing other tests, which may or may not contain more tests and you just keep recursing until you finally hit an actual Test and execute it. With this approach, where do i spin off the threads?

I did a quick hack of the code to test what would happen if I added a check to see if all the Tests in the current TestSuite are 'NUnitTestFixtures" and then executing these in parallel. Psuedo code for TestSuite.RunAllTests:

if (tests.TrueForAll(delegate(Test t) { return t is NUnitTestFixture; })) {
    foreach (Test test in Tests)
        ThreadPool.QueueUserWorkItem (delegate { ProcessTestNormally (test); });
    WaitForAsyncTestsToComplete ();
}
else {
    foreach (Test test in Tests)
        ProcessNormally (test);
}
 

As I'm sure you know, testing for a particular class isn't a good idea. That's especially true at the level of an NUnitTestFixture.

 
However this revealed that the design of how the tests are executing cannot support running tests in parallel. I kept getting errors in the NUnit code caused by the multiple worker threads stepping on each other. So, rather than wasting *too* much time scouting around the code trying to figure out how best to work this out, do you have any suggestions? I'm all ears :)
 

You'd have to show me the specific code, but I can imagine why it's a problem. The Test class only as a synchronous run and

Tests run their contained tests. So while it's possible to create a separate thread - I do it in several cases - you can't easily make that thread run asynchronously because all the callers up the chain are expecting a synchronous response.

 

The TestRunner interface and implementing classes /do/ have an asynchronous entry. The reason I can easily run assemblies

in parallel using the MultipleTestDomainRunner is that it creates a separate runner per assembly. I can also do it with separate

processes if you have selected that option.

 

In NUnit 3.0, the way of running tests needs to change. Recursive descent down the tree of tests is too simplistic for the options

we want to implement. But this is a big change with lots of possible side effects - which is why I have been talking about it for 3.0 only. 

 

So, I see two options - and we could even pick both of them:

1) Implement parallelism at the assembly level in 2.x

2) Work on an improved way of running tests in 3.0 so we can implement it at any level.

 

It's not that fixture level parallelism can't be done in the 2.x architecture, it's just a real pain to do it.

 

Charlie 
 

 Alan.

[Alan McGovern]

Hey.

NUnit doesn't currently guarantee order. I thought you were proposing we guarantee a certain order. 

Oh, sorry. I thought you said that NUnit guaranteed that it ran tests in the order they were in the nunit file. I was just saying that that guarantee could no longer be given with parallel tests. I personally don't think ordering should ever be guaranteed, it's too restrictive a guarantee.

Yeah, I checked out the code earlier and did a test build. However, before that I did a little benchmarking. I wanted to see what the real-world benefit was before I wasted time trying to implement something that gave a negligible boost. The source code for the sample parallel runner is attached to the email. Basically I execute all the NUnit tests in the current assembly in parallel and see how long it takes. If you want to test this code on your own NUnit tests, just compile Main.cs with all your NUnit tests and then execute the resulting console application.

So here's some results from running the 220 odd tests for MonoTorrent:

1 Thread:  16.8 seconds
2 Theads: 11.6 seconds
3 Threads: 10.2 seconds
4 Threads: 9.0 seconds
5 Threads: 9.1 seconds 

 

I saw substantially better improvement when parallelizing at the assembly level - probably due

to lower overhead. 

If i dequeued two/three types at a time for each worker thread, it may improve performance further. It's just a case of figuring out what's the most efficient method of splitting tests up with some careful profiling/timing ;) Either way, we have nearly a 2x improvement in runtime, which is definitely nice.

As I'm sure you know, testing for a particular class isn't a good idea. That's especially true at the level of an NUnitTestFixture.

Yeah, it was just the quickest hack i could think of to try and see what the possibly issues would be for parallelising things. It served it's job well because I found that the rest of the code requires the tests to be synchronous, otherwise nasty things happened.

In NUnit 3.0, the way of running tests needs to change. Recursive descent down the tree of tests is too simplistic for the options

we want to implement. But this is a big change with lots of possible side effects - which is why I have been talking about it for 3.0 only. 

 

Yeah, this is pretty much what i gathered from my look at NUnit 2.x. SO, i figure it'd probably be better just to implement this for 3.x. There's probably not much point in spending ages implementing this for 2.x just so it can be re-implemented for 3.x after only a few releases.

So, I see two options - and we could even pick both of them:

1) Implement parallelism at the assembly level in 2.x

From the sounds of it, this is mostly implemented already. So yeah, this would be a pretty good option to enable.

2) Work on an improved way of running tests in 3.0 so we can implement it at any level.

 

I'm up for that. Is there a 3.x branch yet, or are things mostly in the design/talk phase?

Alan.

[Charlie Poole]

Hi Alan,

 

So, I see two options - and we could even pick both of them:

1) Implement parallelism at the assembly level in 2.x

From the sounds of it, this is mostly implemented already. So yeah, this would be a pretty good option to enable.  

 

It has a few glitches, which is why it's disabled. It doesn't seem as if you would get any benefit from it in your situation, but it might help a few people. There's a #define in AggregatingTestRunner.cs that turns it on if anyone wants to experiment with it. That class is the base used both for running assemblies in separate AppDomains or for running in separate processes, so it affects both.

 

Currently, running the NUnit tests in separate AppDomains gives 22 failures and 22 errors when this is turned on. With

separate processes I get 1 error and 3 failures. I think these are due to problems in the tests themselves and - in the first case - with how the config file is determined. There's a problem with the Gui never being notified that the last test ended, which is understandable since there's no code in AggregatingTestRunner to do it. :-)

 

 2) Work on an improved way of running tests in 3.0 so we can implement it at any level.

 

I'm up for that. Is there a 3.x branch yet, or are things mostly in the design/talk phase?  

 

Not yet. But if you're interested in working on it, you could get started in several ways:

 

1) Modify a private copy of NUnit to add another runner to replace SimpleTestRunner. Rather than relying on TestSuite to handle the recursive descent, it would need to walk the tree of tests. This would, of course require modifying TestSuite as well. As I describe it,

this sounds complicated, reminding me of why I wanted a fresh start with 3.0. :-) 

 

2) Use NUnitLite as a base. It will be the base for the new 3.0 framework and looks a lot like NUnit except it's only one assembly. You might be able to use it for your own tests, in fact. Because though NUnitLite is small, it's probably going to be a lot easier to make the necessary changes. In fact, it merely runs a list of fixtures, just like your example program.

 

NUnitLite doesn't have a gui, but the NUnit gui can be used to run its tests. Of course, when it does that it uses its own logic to run the tests, but it would be quite simple to create an addin that simply delegated all the running to NUnitLite. I haven't looked into this in detail, but it might be the approach to consider and I was already planning to start on 3.0 by merging the NUnit and NUnitLite frameworks into a common codebase.

 

If you want to give tis a try, you'll find nunitlite at http://launchpad.net/nunitlite. The directions for installing Bazaar and getting a copy of the source are the same as those given for NUnit at http://nunit.org/?p=howtos.

 

Charlie

[Alan McGovern]

Out of interest, should i be restricting myself to .NET 1.0/1.1 types, or am I allowed use .NET 2.0, or even 3.0? For the moment I'm going to be coding in 2.0 (so generics), but i can switch that around later if required.

Alan.

[Charlie Poole]

Hi Alan,

 

It depends where you are in the system.

 

NUnit through 2.4.x is all .NET 1.0/1.1/2.0+ compatible.

 

With 2.5, we want to be able to run tests under .NET 1.0/1.1 but the Gui will be 2.5.

 

I think that means the stuff you are interested in will have to somehow work - at least

eventually - under .NET 1.x, since the test runner is always in the same process as

the tests, and the users may want to test under different versions.

 

That's not to say you couldn't start in .NET 2.0, but remember that anything requiring

2.0 will eventually be inside and #if/#endif and we'll have to figure out the #else part.

So I'd avoid using 2.0 features gratuitously. For example, I use generics where they

improve efficiency due to eliminating boxing, but I don't use them (in this app) just

because they are more expressive.

 

If you build with C# 3.0, you should target .NET 2.0. That makes us compatible

with 2.0, 3.0 and 3.5. I haven't tried it with 4.0 yet.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern
Sent: Thursday, December 25, 2008 1:04 PM
To: nunit-...@googlegroups.com
Subject: [nunit-discuss] Re: Running NUnit tests in parallel

[Alan McGovern]

Hey,

I did a bit of prototyping with nunitlite and things are looking promising. I just want to clarify exactly what will/won't be supported, then I'll get you a patch.

* We will only support fixture level parallelism.
* Only fixtures marked with [Parallel] will be executed in parallel.
* NUnit will have a boolean setting (somewhere) which will initially be populated by whether or not [assembly:Parallel] exists. Fixtures will *only* be run in parallel this settings is 'true'. GUIs will be able to flick this option to enable/disable parallel tests.

So, just to clarify:
Should we consider a test marked with parallel safe to execute in parallel with *any* other test or should we consider it as safe to execute in parallel with other [Parallel] tests? I think that a fixture marked with [Parallel] should be taken as a guarantee that the fixture is safe to execute in parallel with any test. That sound ok?

Alan.

[Charlie Poole]

Hi Alan,

 

That sounds cool -  I'd really like to see it when it's ready.

 

Bear in mind that your parallel thing will currently only work when running

tests built as an exe against NUnitLite. NUnit (currently) doesn't run NUnitLite

tests that way, but it could be made to do it.

 

I agree with your interpretation of Parallel. If we want something more controlled,

then we will need some sort of XML specification of what the parallel groups

are - just as pNUnit does it.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern

Sent: Thursday, December 25, 2008 6:19 PM

[Alan McGovern]

Hey,

So here's a provisional patch, mostly to get feedback from yourself (and others) about how to go about this. What I was aiming for here was to minimise changes to the public API, so this may not be the cleanest way to implement these changes. Also, i'm fairly sure my messing with delegates will break .NET 1.1 compatibility, but that can be fixed easily enough if the general idea is solid.

So, the changes are essentially:
1) The addition of the 'Parallel' attribute.
2) TestSuite preprocesses its list of tests into two queues, one for Parallel tests and one for Sequential tests.
3) Multiple threads will operate on the Parallel tests and a single thread will operate on the Sequential tests. Both these sets will run at the same time.
4) The number of parallel threads to use is read from GlobalSettings.ParallelCount. A value of zero or less disables threaded running.
5) A TestSuite will always default to running sequentially. You must change the setting of TestSuite.ParallelTests (maybe ParallelThreads would be better?) to enable additional threads. The reason for this is that TestLoader.Load (assembly) will set this value as it creates the root-level test suite. Other TestSuite instances will *not* be root-level and so setting ParallelThreads on these will result in test-level parallelism, not fixture-level parallelism.

So, patch is attached, it works (as far as i can tell). I hate bzr in windows ;) I'm also hating bzr in general because it doesn't understand that "bzr revert" in a specific directory means revert in the directory and subdirectories only - it decided to revery *every* change in every directory. Luckily i had created a diff beforehand, just in case something crazy like that happened ;)

For a cleaner overall approach, one which is less fragile, I think a refactor along the lines of this would be in order:

public ITestSuite : ITest
{
    ArrayList TestFixtures { get; }
}

public SequentialSuite : ITestSuite
{

}

public ParallelSuite : ITestSuite
{
    int Threads { get; set; } // The number of threads to use to run the tests
}

public ITestFixture : ITest
{
    ArrayList TestCase { get; }
}

public TestCase : ITest
{
   
}

Then TestLoader would become:
ITestSuite TestLoader.Load (Assembly)
ITestSuite TestLoader.Load (Type, String)

etc.

The result would be that ITestSuite is guaranteed to be your top level node. This will contain a list of TestFixtures which contain the Tests to be executed. The ITestSuite is the one which controls how the TestFixtures are executed. It can execute them in parallel, or in series.

Similarly, later on you could implement ITestSuite level parallelism by creating:

public class ParallelSuiteTester : ITest
{
    public ArrayList TestSuites { get; }
}

However, these changes would be fairly major and API changing, and I'm in no way familiar enough with NUnit to know whether that'd be a good or bad thing ;)

Alan.

[Charlie Poole]

Hi Alan,

 

Thanks! This goes into my queue right after getting the 2.5 beta 2 out.

 

Preliminary comments, from reading your note and the patch:

1) I don't quite get the purpose of giving the user two ways to set ParallelCount -

through an assembly attribute and through GlobalSettings.

2) We do need to figure out whether parallel is a good idea on all the platforms

that NUnitLite supports - maybe some netcf folks will comment on this.

 

I'll get what you have in place and then look at possible refactoring.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern

Sent: Friday, December 26, 2008 6:40 AM

[Charlie Poole]

Hi Alan,

So, patch is attached, it works (as far as i can tell). I hate bzr in windows ;) I'm also hating bzr in general because it doesn't understand that "bzr revert" in a specific directory means revert in the directory and subdirectories only - it decided to revery *every* change in every directory. Luckily i had created a diff beforehand, just in case something crazy like that happened ;)
 

Of course anything working differently from what you're used to is likely to be confusing, but Bazaar's behavior is documented in the help and seems to me to make more sense than being directory oriented. Unless you specify othewise, a revert removes your last commit, whatever it was.

 

Charlie 

[Alan McGovern]

On Fri, Dec 26, 2008 at 6:26 PM, Charlie Poole <cha...@nunit.com> wrote:

Hi Alan,

 

Thanks! This goes into my queue right after getting the 2.5 beta 2 out.

 

Preliminary comments, from reading your note and the patch:

1) I don't quite get the purpose of giving the user two ways to set ParallelCount -

through an assembly attribute and through GlobalSettings.

Yeah, this is a bit of an issue. Actually, I think a better way to work this would be to just use [Parallel] as a parameterless attribute and automatically choose the number of threads to use, say (1.5 * LogicalCores + 1). This way we can ditch the setting in GlobalSettings and simplify the [Parallel] attribute. Problem solved.

2) We do need to figure out whether parallel is a good idea on all the platforms

that NUnitLite supports - maybe some netcf folks will comment on this.

If you don't want parallel execution on certain platforms, the threaded running can be disabled either via ifdefs or runtime checking. It depends on what's required.

Thanks,
Alan.

[Charlie Poole]

Hi Alan,

1) I don't quite get the purpose of giving the user two ways to set ParallelCount -

through an assembly attribute and through GlobalSettings.

Yeah, this is a bit of an issue. Actually, I think a better way to work this would be to just use [Parallel] as a parameterless attribute and automatically choose the number of threads to use, say (1.5 * LogicalCores + 1). This way we can ditch the setting in GlobalSettings and simplify the [Parallel] attribute. Problem solved.
 

I think you're right. Being capable of parallelization is a property of the particular tests while the number of parallel threads is

a property of the installation. Ergo, [Parallel] belongs in the tests and MaxNumberOfParallelTests should be an NUnit setting with

a default such as you suggest, but changeable from the Settings dialog.

 

Charlie

[Sébastien Lorion]

Why not simply use the .NET thread pool which we can already configure ?

Sébastien

[Alan McGovern]

Hi,

Using threadpool threads is an option, though configurability is not one of the reasons why using a threadpool thread would be better. You have to bear in mind that the application your testing and your nunit tests share the same threadpool, so any changes you make to the threadpool can easily adversely affect the application you're testing.

I don't think it makes too much difference whether a threadpool thread is used or a standard Thread object. I went with a standard Thread object as it simplified the logic required to wait for the threads to exit - I could just call Thread.Join. So unless there's a specific reason why a Threadpool thread would be better, is there any reason to add the additional complexity? Sure, it's only 5-6 lines of extra code, but it's 5-6 lines which don't need to be there ;)

Alan.

[Charlie Poole]

NUnit doesn't use a threadpool for test threads for another reason: sometimes the thread needs to

be an STA thread. NUnitLite doesn't have that feature... yet.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern

Sent: Friday, December 26, 2008 5:10 PM

[Sébastien Lorion]

Well, if you would be creating a thread for each test instead of each fixture, you would need a thread pool for sure (possibly another than the one from .NET for the reasons you gave and more), but for test fixtures only, creating a new thread for each seems ok. I was thinking a thread pool would simplify the thread creation management that you were mentionning (ie calculating number of threads, etc.), but if it does not, well ok. Anyway, that's something that can be done later on without impact on the api/behavior.

Sébastien

[Alan McGovern]

Hey,

On Sat, Dec 27, 2008 at 4:04 AM, Sébastien Lorion <sebastie...@gmail.com> wrote:

Well, if you would be creating a thread for each test instead of each fixture, you would need a thread pool for sure (possibly another than the one from .NET for the reasons you gave and more), but for test fixtures only, creating a new thread for each seems ok.

Funnily enough - that'd be ridiculously slow ;) The initial prototype of threaded test running did something along the lines of:

TestResult result = new TestResult (this);
foreach (Type testFixtureType in allfixtures)
    TheadPool.QueueUserWorkItem (delegate {
        TestResult r = ExecuteTest (testFixtureType);
        result.AddResult (r);
    });

There was also a little book-keeping so only N threads would be executing tests at the same time. This was horribly slow. What was happening is that you had to
A)  Find a threadpool thread
B) Wake it up
C) Make it execute the test
D) Return it to the pool
E) Let it fall asleep

Threads ended up spending most of their time waking up or falling asleep and not executing tests. A quick rewrite to a 'pull' model instead of a 'push' model solved that pretty quickly. Now each thread pulls a type from a shared Queue, executes it, then pulls the next type from the queue. These threads are created once per testsuite, not per testfixture, so if you're executing 50 TestFixtures on a dual core, you'll end up creating 4 threads at the start and each thread will execute about 50/4 tests. As these are once off expenses, the saving from uses a pre-existing thread in the threadpool wouldn't be huge. Combine that with the need to enforce a STA thread (not required yet though) and the threadpool isn't quite as inviting.

Hope that helps explain the reasoning why 'new Thread' isn't that much worse than threadpools.

Alan.

[Charlie Poole]

Hi Alan,

 

That's good info and it confirms what I've observed as well. When Olof first suggested

that we implement Timeout by using a thread per test I balked because it sounded as

if it would be too costly. It turned out that creating a thread per test case added 1 or 2

milliseconds per test case, which isn't that bad if you really need to have a global

per-case timeout value.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern

Sent: Saturday, December 27, 2008 2:19 AM

[Sébastien Lorion]

You are comparing apples with oranges here... the pull model is also possible with a threadpool. For my own curiosity, I implemented a version of ExecuteTests using threadpool and it is faster, partly because there is less Queue bookeeping. If I use a Stack instead of a Queue, either in my implementation or yours, many tests are failing, not exactly sure why.

Anyway, the STA thread requirement and the fact that nunit is sharing the pool with the application being tested make the issue void.

Btw, in your implementation, it would be faster to not enqueue parallel tests into the sequential queue at the start if ParallelTests == 0, but to execute threadProc another time before or after the sequential tests:

if (ParallelTests <= 0)
  threadProc(parallelTests);

Sébastien

[Alan McGovern]

Hey,

You are comparing apples with oranges here... the pull model is also possible with a threadpool.

And all you save with this version is the once-off costs of initialising the 3 or 4 threads required to run the tests. I did say before that threadpool threads could easily be used with this code, it just adds 6-7 extra lines of book keeping because you can't use Thread.Join to wait for the threaded tests to complete.
 

For my own curiosity, I implemented a version of ExecuteTests using threadpool and it is faster, partly because there is less Queue bookeeping.

The only performance difference would be if "new Thread ().Start ()" is slower than using "ThreadPool.QueueUserWorkItem" and the associated bookkeeping. As the costs are once off, i figured they'd be relatively minor. Over a small set of tests, the thread starting costs would probably be noticable. Over a 17-20 seconds testsuite, they'd be negligible.

The exact same bookkeeping for the Queue is required in both implementations though, otherwise it's not threadsafe ;)

If I use a Stack instead of a Queue, either in my implementation or yours, many tests are failing, not exactly sure why.

That's interesting. They should be exactly equivalent. I'll test that locally. Could it be that the tests you're running are not parallelisable and are order dependent?  

Anyway, the STA thread requirement and the fact that nunit is sharing the pool with the application being tested make the issue void.

Btw, in your implementation, it would be faster to not enqueue parallel tests into the sequential queue at the start if ParallelTests == 0, but to execute threadProc another time before or after the sequential tests:

if (ParallelTests <= 0)
  threadProc(parallelTests);

Very good point. That's definitely a good optimisation.

Alan.

[Sébastien Lorion]

On Sat, Dec 27, 2008 at 3:17 PM, Alan McGovern <alan.m...@gmail.com> wrote:

Hey,

You are comparing apples with oranges here... the pull model is also possible with a threadpool.

And all you save with this version is the once-off costs of initialising the 3 or 4 threads required to run the tests. I did say before that threadpool threads could easily be used with this code, it just adds 6-7 extra lines of book keeping because you can't use Thread.Join to wait for the threaded tests to complete.
 

For my own curiosity, I implemented a version of ExecuteTests using threadpool and it is faster, partly because there is less Queue bookeeping.

The only performance difference would be if "new Thread ().Start ()" is slower than using "ThreadPool.QueueUserWorkItem" and the associated bookkeeping. As the costs are once off, i figured they'd be relatively minor. Over a small set of tests, the thread starting costs would probably be noticable. Over a 17-20 seconds testsuite, they'd be negligible.

The exact same bookkeeping for the Queue is required in both implementations though, otherwise it's not threadsafe ;)

If I use a Stack instead of a Queue, either in my implementation or yours, many tests are failing, not exactly sure why.

That's interesting. They should be exactly equivalent. I'll test that locally. Could it be that the tests you're running are not parallelisable and are order dependent?  

 

I am running the NUnitLite unit tests, just like you did. I asked myself the same question, but I did not investigate further.

[Alan McGovern]

Hey,

equivalent. I'll test that locally. Could it be that the tests you're running are not parallelisable and are order dependent?  

 

I am running the NUnitLite unit tests, just like you did. I asked myself the same question, but I did not investigate further.

Turns out it's an obvious solution. Some of the NUnit lite tests are dependent on the order in which TestCases are executed inside a TestSuite. If you use a stack instead of a queue, you reverse the order of the TestCases as well as the order in which the TestSuites are executed.

5) CanLoadAndRunSuiteFromSuiteProperty (NUnitLite.Runner.Tests.TestLoaderTests.CanLoadAndRunSuiteFromSuiteProp
erty)
  String lengths are both 80. Strings differ at index 10.
  Expected: "<MyTests:<One::Success><Two::Success><TheTest::Failure><Three..."
  But was:  "<MyTests:<Three::Success><TheTest::Failure><Two::Success><One..."

As can be seen, the tests executed in the opposite order to that which the actual test expected, hence the failure. Not a bug as such.

Alan.

[Charlie Poole]

Yes, some of the tests in NUnitLite rely on the (non-guaranteed) ordering. That's

over-specification, of course, but it made life a bit easier.

 

The same thing could happen if one thread happened to finish ahead of another,

so those tests should be non-parallelizable.

 

Charlie

---

From: nunit-...@googlegroups.com [mailto:nunit-...@googlegroups.com] On Behalf Of Alan McGovern

Sent: Saturday, December 27, 2008 12:34 PM

[Olof Bjarnason]

2008/12/27 Charlie Poole <cha...@nunit.com>:

> Hi Alan,
>
> That's good info and it confirms what I've observed as well. When Olof first
> suggested
> that we implement Timeout by using a thread per test I balked because it
> sounded as
> if it would be too costly. It turned out that creating a thread per test
> case added 1 or 2
> milliseconds per test case, which isn't that bad if you really need to have
> a global
> per-case timeout value.

Yeah I simply wrote a small test program to test how fast
creating-running Threads under .NET was on my 2 year old PC, and it
was 1.7 ms IIRC per thread.

If there is a way to specify the Timeout-value for all tests of a
nunit-console run, it is quite simple to check how much the overhead
is in your case. (Measuring with and without the that flag, dividing
by the number of tests run..)

[Charlie Poole]

Hi Olof,

> If there is a way to specify the Timeout-value for all tests
> of a nunit-console run, it is quite simple to check how much
> the overhead is in your case. (Measuring with and without the
> that flag, dividing by the number of tests run..)

There is such an option: /timeout=<value in milliseconds>

Charlie

[Olof Bjarnason]

I'd be interested in hearing the overhead values for other setups than
mine! Care to share such a measure from one of your 1000-test projects
Charlie ? :)

2008/12/28 Charlie Poole <cha...@nunit.com>:

[Charlie Poole]

Hi Olof,

I just ran the loadtest-assembly.dll both ways. It has 1000 empty
tests. Results were...

No Timeout 1046 milliseconds
with Timeout 2828 milliseconds

Based on this, I'd say the overhead per empty test is about
1 ms, with almost 1.8 added if we create a thread. For very
small tests, it's a significant percentage increase, but it's
probably not needed for very small tests anyway.

Personally, I see using this feature as follows:

1. Mark fixtures or cases individually for normal development
in the Gui. Only mark those that have some expectation of
timing out, like tests waiting for a connection.

2. Use a global timeout when running the continuous or
nightly build, with a fairly big setting. The individual
settings would override the global setting, but that
can be taken into account when specifying the values.

Charlie

> -----Original Message-----
> From: nunit-...@googlegroups.com
> [mailto:nunit-...@googlegroups.com] On Behalf Of Olof Bjarnason
> Sent: Sunday, December 28, 2008 1:16 PM
> To: nunit-...@googlegroups.com
> Subject: [nunit-discuss] Re: Running NUnit tests in parallel
>
>

[Olof Bjarnason]

2008/12/28 Charlie Poole <cha...@nunit.com>:
>
> Hi Olof,
>

> I just ran the loadtest-assembly.dll both ways. It has 1000 empty
> tests. Results were...
>
> No Timeout 1046 milliseconds
> with Timeout 2828 milliseconds
>
> Based on this, I'd say the overhead per empty test is about
> 1 ms, with almost 1.8 added if we create a thread. For very
> small tests, it's a significant percentage increase, but it's
> probably not needed for very small tests anyway.
>
> Personally, I see using this feature as follows:

I'm not sure if I'm slow tonight, but for me, the primary use case of
Timeout is:

0. More user-friendly development of methods with loop-logic
(you don't have to ps-kill or taskmanager-kill the nunit-console
application in case you mistankenly [or by free will] create an
infinite loop in your class-under-test)

[Charlie Poole]

Hi Olof,

> I'm not sure if I'm slow tonight, but for me, the primary use
> case of Timeout is:
>
> 0. More user-friendly development of methods with loop-logic
> (you don't have to ps-kill or taskmanager-kill the
> nunit-console application in case you mistankenly [or by free
> will] create an infinite loop in your class-under-test)

In that case, I imagine you would use the global setting in your
normal development work. I wouldn't normally do that, but I
could see turning it on in case of having a bad day. :-)

Charlie