Test case run order

[Hanek]

I went through posts concerning test execution order, but didn't find
info if there is already some feature implemented. The discussion
seems to have been going on for at least a couple of years now.

I am working with Selenium WebDriver end-to-end UI test cases written
in C#, which are run using NUnit framework. (Not unit cases as such)

These UI tests certainly depend on order of execution, as other test
needs to add some data as a precondition. (It is not feasible to do
the steps in every test)

Now, after adding the 10th test case, i see NUnit wants to run in this
order: Test_1 Test_10 Test_2 Test_3 ..

So i guess i have to too alphabetisize the test case names for now,
but it would be good to have this feature of controlling execution
order added to NUnit. (I am using now NUnit 2.6)

Is the execution order feature coming anytime soon? (Or should i start
to hack the source...?)

Br,
--Hannu

[John Brett]

<snip/>

> So i guess i have to too alphabetisize the test case names for now,
> but it would be good to have this feature of controlling execution
> order added to NUnit. (I am using now NUnit 2.6)

Hi,

Sorry for butting in, but whilst I understand the constraints you
have, I don't
think that this should go into NUnit. NUnit's focus is that of a unit
testing
platform, and it would be inappropriate for unit tests to be coupled
or
order-dependent.

Having used NUnit in the past for integration and system level tests,
the
dependencies between tests kept biting me, and I ended up having to
code the tests to be as independent as possible. It's the only way to
have
reliable and stable test runs.

I think adding any feature to NUnit to make it easy to order execution
would lead to bad habits and therefore to lots of unhappy users.

Having said that, I suspect that you could use a TestCaseSource
(http://www.nunit.org/index.php?p=testCaseSource&r=2.6)
to orchestrate the order of execution of test cases.
Alternatively, if you have tests that have to run from start to end in
order to
be valid, you could just write the one uber-test, or a parent test
that executes
individual test methods explicitly.

> Is the execution order feature coming anytime soon? (Or should i start
> to hack the source...?)

Ouch :-(

John

[Charlie Poole]

Hi John and Hannu,

What John says is quite right as far as the historical intent of NUnit
as a framework for creating independent unit tests. My plans for the
future are slightly different...

NUnit 3.0, IMO, should give more support to other kinds of tests (e.g.
integration tests, web tests) while not doing anything to degrade its
support for unit testing. When I talk of "not degrading" I include the
idea that we should not add a lot of general features that could
confuse a writer of unit tests.

My own idea of how to do this - subject to suggestions from the
community, of course - is that there should be a different sort of
test class (i.e. not a Test Fixture), which is intended for use with
the sort of higher level tests that some people use with NUnit. Within
such a class, I imagine that various methods would be marked as
"Steps" in the test, with their order clearly specified.

An approach like this - it seems to me - would keep the user interface
for writing unit tests clean, while allowing an entirely new set of
features to be supported for higher level tests. Of course, much of
the underlying code could be shared between the two implementations.

Thoughts?

Charlie

> --
> You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
> To post to this group, send email to nunit-...@googlegroups.com.
> To unsubscribe from this group, send email to nunit-discus...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nunit-discuss?hl=en.
>

[John Brett]

Hi Charlie,

This broadly gets my vote.

The primary differences I see between the clean world of unit testing
and mucky world of integration/system testing are:

* ordering of operations

* optional tolerance of intermediate operations failing

* retry logic

* common error handling

None of these I'd want to see in the unit testing side of things (TestFixture),
but would be useful for other types of test.

John

[Charlie Poole]

Hi John,

Nice summary of the differences. That's pretty much as I see it.

Charlie

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/nunit-discuss/-/rWgltKERIccJ.