Parallelizable tests within non-Parallelizable SetupFixtures.

[Paul Hicks]

I have two suites-of-suites (I'll call them "areas"), each of which require destructive setups (deleting common data from a database).
All the tests at all levels within each area can run in parallel with all the other tests in the area. All the tests conflict with all tests in the other area.
What attributes do I use, and where, in order to achieve maximum parallelization?

I have created SetupFixtures in each of the two namespaces Project.Area1 and Project.Area2. Each of these has the Parallelizable(ParallelScope.Children) attribute. I expect that to allow all the tests in one area to run in parallel without any tests in the other area running; after that area's tests finish, all the tests in the other area should run in parallel. What I'm observing is that all tests in both areas are running in parallel, and the destructive setups are deleting in-use data from the other area of tests. Is this expected? What do I do to fix it?

[Charlie Poole]

This is covered by issue #1474. Issue #1239 may be a symptom of the
same problem.

SetUpFixtures attributed as you have done will not run in parallel.
However, this says nothing about the children of those fixtures. What
happens is that the two SetUpFixtures are placed in a non-parallel
queue. They run sequentially. As each completes it's OneTimeSetUp, it
adds it's child fixtures to a queue - in this case, a parallel queue
with multiple workers. They are free to run in any order. Put in other
words, NUnit does not consider a SetUpFixture to be "running" in
between it's OneTimeSetUp and OneTimeTearDown. The OneTimeTearDown,
btw, runs on the same thread as the last fixture that finishes
running. This is also a potential problem, although it's less obvious
in the case of TearDowns.

Issue #1474 is in design status, which means "We have to figure out
how to do this." I just increased its priority.

As a workaround, since you only have two "areas", I suggest putting
them in separate assemblies and running both of them serially. You can
either run NUnit twice or use --process:Separate so that they run
sequentially. Of course, depending on what your tests are doing, you
may even to be able to run in parallel using two processes.

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Paul Hicks]

Thanks for that, that all makes sense. In the short term, would this alternative work for me? I could create one base class per area, make them both TestFixture and Parallelizable(ParallelScope.Children), and have all TestFixtres in each area extend the correct base class. I could even repurpose the SetupFixture for that.

I can put a TODO on them both to check out 1474 every few weeks.

[Charlie Poole]

Hi Paul,

Nope, because derived fixtures are not "children" of the base fixture.
If you don't want to use separate assemblies, you could use separate
runs. Exclude one of the SetUpFixture namespaces in one run and
include it in the other. Alternatively, use a synchronization object.
Acquire it in your OneTimeSetup and free it in your OneTimeTeardown.

If you have any ideas about how to do the enhancement, please comment
on #1474. The key issue is that _some_ SetUpFixtures need to work as
you want but others are just fine with the way things are done now. It
all depends on what you are doing in the setup fixture.

See also issue #165 (that's old!) and the ExclusionGroup writeup on
this page:https://github.com/nunit/docs/wiki/Framework-Parallel-Test-Execution-Spec

[Paul Hicks]

What things are children of other things? What will a [TestFixture, Parallelizable(ParallelScope.Children)] not run at the same time as?

[Paul Hicks]

The docs say "ParallelScope.Children indicates that the descendants of the test may be run in parallel with respect to one another." Perhaps that needs a few more words to explain what a descendant is in this case?

[Charlie Poole]

Hi Paul,

Well, I suppose the definition seemed obvious from a certain
perspective - that of the NUnit developers - but it needs to be made
obvious to users. :-)

Children _should_ mean tests contained within other tests. In other
words, tests that would be displayed in the old NUnit GUI under those
other tests. Or tests included under the XML element of another test
in the result file.

WRT SetUpFixtures, this is broken. TestFixtures in a namespace are
"children" of the SetUpFixture for that namespace. What you want to do
should work. However, we only enforce the "Non-parallel" restriction
for the duration of the OneTimeSetUp method. That's what is messing
you up. However, I believe it's a feature for some people, so that
makes it hard to decide what to do with it. We may need to have some
way of designating which SetUpFixtures work one way and which work the
other way.

Nested fixtures have never been treated as children. You'll see them
displayed in the GUI completely independently and reported in the XML
the same way. We have, in fact, an issue to treat them as children,
which would be a major change in the way NUnit has worked for the past
fifteen years or so.

Inherited fixtures are not children and will never be. Inheritance is
just a way to put some common tests into different fixtures.

You're right, we shuld explain this better in the docs.

To your first question... the TestFixture will _not_ run at the same
time as any other test fixture.

Charlie