Can OneTimeSetUp be non-async?

[Paul Hicks]

The wiki says "OneTimeSetUp methods may be async if running under .NET 4.0 or higher", and indeed I'm seeing exactly that.
Usually it's no problem, but I'd like to be run some OneTimeSetUps and OneTimeTearDowns synchronously: specifically, the ones in Fixtures that are in ApartmentState.STA. I'm finding that the OneTimeSetUp of my STA tests is overlapping with the SetUp of my MTA tests, and it's causing issues for me.

My workaround for now is to change my 10-test suite into a 10-assertion test and use SetUp instead of OneTimeSetUp. Less than ideal.. :(

[Paul Hicks]

Update: my workaround doesn't work either :( SetUp is also asynchronous.
Putting all tests in the STA apartment works though.

Darn.

[Oskar Berggren]

Doesn't the wiki intend to say that if you want, you MAY use the async keyword on OneTimeSetUp if running under .Net 4.0?

I didn't look at the docs just now, but I find it hard to believe that anything could ever work properly if NUnit suddenly "at random" runs things asynchronously without being asked to.

/Oskar

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

[Oskar Berggren]

It means that unless you've added the async keyword, there is in fact something else going on. Perhaps you setup method is itself calling something that executes asynchronously?

/Oskar

[Charlie Poole]

Oskar's interpretation is correct. Tests, SetUp, OneTimeSetUp, etc.
have been standard synchronous methods for a long time. When we added
support for async, we used the phrase "may be async" to indicate the
new feature.

Of course, your method may be required to be async for reasons that
have nothing to do with NUnit... generally, because it calls other
async methods. Make your methods async where the language requires it
only.

Charlie

[Paul Hicks]

Perhaps I've used the wrong terminology.

I see behaviour when I'm debugging:
- breakpoint at top of OneTimeSetUp in class A (ApartmentState.STA) is hit
- I step
- call stack jumps to breakpoint at top of SetUp in class B (ApartmentState.MTA) is hit
- I step a few times until conflict happens, throwing exception in the SetUp method
- call stack jumps to line after breakpoint at top of OneTimeSetUp in class A, as if class B hadn't been involved
- hell breaks lose and all tests fail.

Does that give you any hints as to what might be going on?

Cheers!

[Charlie Poole]

When debugging, it's useful to ensure that only one test is running at
a time. Do you have parallelism enabled?

In that case, there are several techniques that will help in debugging
under nunit3-console...

1. If running under the IDE, you can use --inprocess and --workers:0
so that you can easily debug and only on test runs at a time.\

2 If your test cannot run in process (e.g. You want to run under .NET
2.0 and the gui is running under .NET 4.5) start nunit3-console
outside of the IDE using --debug and --workers:0. You will be prompted
to select a debugger before the agent begins to execute tests in it's
separate process.

In either case, --workers:0 ensures that one test runs at a time. Note
that --workers:1 will do the same thing, but will set up additional
queues for running parallel and non-parallel tests, which is
semantically equivalent but involves unnecessary overhead.

Charlie

[Paul Hicks]

Actually I'm debugging locally to see why I get conflicts when running on the CI machine.
Currently I'm using --workers:1 on CI (I'll change that to :0, thanks for the tip), but I do intended to move to the default number of workers once some threading issues in our test harnesses are sorted. We want our tests to be as parallel as feasible.
I'm seeing the same issues when running from nunit3-console as I'm seeing from the IDE.

So assuming that I'm not debugging, is there a reason why OneTimeSetup in a STA class would run at the same time as SetUp in an MTA class? What can I do to avoid having anything STA run at the same time as anything else in any apartment? Can I annotate the SetUp / OneTimeSetUp methods, perhaps?

[Charlie Poole]

Hi Paul,

OK... to distinguish --workers:0 from --workers:1, we have to get into
internals. Everyone else, stop reading. :-)

Tests are run like shifts in a factory. We have three shifts. First
shift runs all parallelizable tests in an MTA using as many queues as
you specify **plus** any parallelizable STA tests in a separate queue.
Second shift runs non-parallelizable tests in an MTA. Third shift runs
non-parallelizable tests in the STA. First shift has n queues, second
and third have one each. By definition, shifts don't overlap. If you
could prove that was happening, it would be a bug. :-)

What do you mean when you say same issues when running from
nunit3-console as from the IDE? What I meant was running
nunit3-console either from the IDE or externally. If we try to solve a
problem involving multiple runners, like our adapter or the R# runner,
we are going to get confused.

To answer the specific question, if a class running in the STA is
marked as parallelizable, that means it can run in paralllel with
other tests in the STA. That will happen even if you specify
--workers:1, since an extra queue is created for STA tests, but not if
you use --workers:0 which disables the entire queuing mechanism and
simply runs tests directly like NUnit V2.

Charlie

[Charlie Poole]

Correction to my last paragraph:

"if a class running in the STA is marked as parallelizable, that

means it can run in parallel with other tests running in the
==>MTA<=="

Obviousl, two tests can't run in parallel if both need the STA.

Charlie

[Paul Hicks]

Aha. This is my problem. I hadn't realized this:

On Friday, 15 January 2016 11:13:23 UTC+13, charlie wrote:

"if a class running in the  STA is marked as parallelizable, that
means it can run in parallel with other tests running in the
==>MTA<=="

 
So my follow-up question is: if I've got "[assembly: Parallelizable(ParallelScope.Fixtures)]", can I mark certain test suites as being non-parallelizable? I've been using ApartmentState.STA under the impression that tests using it would be run in the 3rd shift, but I'm not doing anything else. My aim is to have hundreds of suites run in parallel and without boilerplate annotations on each one, but a few certain suites to run afterwards, one at a time, using annotations or similar to mark them. Can this be achieved easily?

[Charlie Poole]

Yes, you can override the gloabal setting on any individual fixture.

Other than debugging, do you see any problem with this default behavior?

Charlie

[Paul Hicks]

On Friday, 15 January 2016 11:32:45 UTC+13, charlie wrote:

Yes, you can override the gloabal setting on any individual fixture.

Other than debugging, do you see any problem with this default behavior?

No, the problem was that I didn't realize that STA could ever run in parallel with anything else; I assumed that it was like Parallelizable(ParallelScope.None).

I have a few custom attributes that currently extend ApartmentAttribute and force the annotated fixture into STA. So I'd like to change those to force the annotated fixtures into ParallelScope.None. However Parallelizable is sealed. Can I achieve the same result by implementing PropertyAttribute and IApplyToContext and applying ParallelScope.None to the TestExecutionContext? Is there anything special about Parallelizable that I would lose by doing this?

I'll give it a go anyway and see if it works.  Thanks for taking the time to explain all this, it's been very helpful.

 

[Charlie Poole]

In general, you are safer creating your own attributes that implement
extensibility interfaces rather than extending NUnit attributes. We
might make some minor change to an attribute's behavior in the future.
Properties attribute is an exception, since it is designed to be
inherited.

So, yes, you should create an attribute that either extends
PropertyAttribute or implements IApplyToTest. That will take care of
making the __current__ test non-parallelizable. If you also want to
make contained tests non-parallelizable, then you should implement
IApplyToContext because that's how the ParallelScope is passed
downward. In your example, IApplyToContext is probably irrelevant, but
it could be relevant in some future scenario.

Charlie

[Paul Hicks]

Thanks Charlie,

I went with IApplyToContext, both because I prefer to annotate the fixture rather than individual tests, and because we have data-driven tests: a single attribute at the fixture level now adds ParallelScope.None to all tests in the fixture, whether they're a [Test], [TestCase] or [TestCaseSource]. I've got everything working on CI now, thanks for all your help!