Ability to build test suite - Nunit 3.10

[travis.n...@gmail.com]

Hi everyone, I've moved to C# for a test project. Apparently, Nunit is new to me and I want to port some home-grew feature from previous test framework I've worked before.

I wonder that is Nunit has public API for doing this?

Here is my scenario:

Objective: I want to select and run a subset of test of my assembly at runtime. I am not familiar with Nunit terminology but I often called it as test suite.

I can provide more detail as below:

Test assembly structure:

- We have multiple test class with [TestFixture] and [Test] appropriately. Here I list two test classes as my simple example.

Test class 1:

[TestFixture]
public class FeatureA {
    [Test]
    public void TestMethodAOne() {
    }

    [Test]
    public void TestMethodATwo() {
    }
}

Test class 2:

[TestFixture]
public class FeatureB {
    [Test]
    public void TestMethodBOne() {
    }

    [Test]
    public void TestMethodBTwo() {
    }
}

Test suite definition is defined and changed every time we run test. Those definitions are stored outside the code(currently I used a database to track content of each execution).

Here are procedure of two executions:

Execution 1:

1. Define FeatureA/TestMethodAOne and FeatureB/TestMethodBOne as execution definition.

2. Trigger test run. Here is where I expect do build a test suite. I would like to have API from Nunit to look at my execution definition at step 1 then run it.

Similarly, next execution, I have a different definition: FeatureA/TestMethodATwo and FeatureB/TestMethodBTwo.

For execution definition, I intend to store test class name and test method name only. I would rather using those names along with Nunit APIs for building test suite rather than using a public API to find test within assembly.

As a newcomer to Nunit, I really appreciate any suggestion and quick hint for implementing my custom feature for my test framework.

Cheers,

Travis

[travis.n...@gmail.com]

Or perhaps I have to prepare test filter with appropriate xml from execution definition. For example, I may draft the filter for my above example as below:

For execution 1:

<filter>

<and>

<namespace>FeatureIntegration.Test.Script</namespace>

<or>

<class>FeatureIntegration.Test.Script.FeatureATest

<method>TestMethodAOne</method>

</class>

<class>FeatureIntegration.Test.Script.FeatureBTest

<method>TestMethodBOne</method>

</class>

</or>

</and>

</filter>

Similarly, for execution 2:

<filter>

<and>

<namespace>FeatureIntegration.Test.Script</namespace>

<or>

<class>FeatureIntegration.Test.Script.FeatureATest

<method>TestMethodATwo</method>

</class>

<class>FeatureIntegration.Test.Script.FeatureBTest

<method>TestMethodBTwo</method>

</class>

</or>

</and>

</filter>

[wayne vetrone]

Would something like this work.

Query your database for the list of tests for TestSuite Foo and output it to a temp file.  Pass the temp file to NUnit Console 

Rinse and repeat

Wayne

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.
To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

[Nguyen Travis]

Thanks Wayne,

With your suggestion, I think we increase more dependencies for test project. As I understand that we have main program of test project looking for test list and turn this into a temp file at build agent. Next step is having a task of PowerShell to trigger desired test execution by NUnit Console. That's is also one of my considerations but I haven't tried Nunit console on VSTS build agent.

However, I think it's just way better if we can construct and run a test suite at runtime.

[Charlie Poole]

Hi Travis,

I understand you want some way to maintain a database of test selections outside of NUnit and to then be able to run a particular selection on command. Since you say you are new to NUnit, it might be best to first understand how NUnit natively allows tests to be selected.

In the following explanation, I assume you are running nunit3-console.exe, since that is the runner that makes use of all of NUnit's capabilities through it's command-line options. In that case, you can define what tests to run by

1. Selecting one or more assemblies. (This is probably obvious, but I wanted to list it to be complete.)

2. Selecting "tests" by name. In this context, tests may be namespaces, test fixture classes, individual methods or individual methods with specific arguments. In your example, if you specified "Some.Namespace.FeatureA", then all the tests in FeatureA would be run. If you specified "Some.Namespace.FeatureA.TestMethodAOne" then only that single test would be run. Using the console runner, you can specify a single "test" by use of the --test option. Perhaps more relevant for you is the --testlist option, which specifies a text file containing the names of tests, one per line.

3. Selecting tests by category. If it's relatively simple to view groupings of tests across multiple fixtures as being in certain categories, you can assign them using the CategoryAttribute and then select those categories to run using the --where option.

4. More general selection logic using Test Selection Language. (This overlaps method 3, since you use TSL to select categories as well.) The syntax of TSL is described in the documentation wiki. You specify it as expressions after the --where option. It is quite rich. If you have a lot of long expressions to use, the @file notation allows you to save a set of options, including the --where option, in a file. Because the @file notation is intended to be quite general, it has no specific interpretation for the --where option. You specify the options using the same syntax that would be used on the command-line, with the only difference being that newlines are allowed and treated as white space.

With the problem you have described, my tendency would be to create a simple way to create and edit testlist files (item 2) and inject them into the command-line by name.

Charlie

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.

[wayne vetrone]

>>  With the problem you have described, my tendency would be to create a simple way to create and edit testlist files (item 2) and inject them into the command-line by name.

This was basically the solution we have gone with (although we might change in the future).

We have saved into a database all TestCases and their Categories.  Additionally we saved run time of each TestCase.   

From that point, our build system (Jenkins) can slice and dice any combination of Tests, output them to a file and send them to the console runner.  

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

[travis.n...@gmail.com]

Thanks Charlie, I can now make the feature works exactly we want with Nunit console. However, just in case I want to be independent of Nunit Console, do you think that the effort is too much duplicate with what have been completed by Nunit Console?

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

To post to this group, send email to nunit-...@googlegroups.com.
Visit this group at https://groups.google.com/group/nunit-discuss.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "NUnit-Discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discus...@googlegroups.com.

[Charlie Poole]

It shouldn't be hard to duplicate this in any program that runs tests using the NUnit engine. All the pieces needed are available from the engine. If you look at the nunit console code, you'll see that it does very little by itself.

If you want this ability in one of the other NUnit-supported runners, file an issue. If you want it in a program of your own, then use the same APIs as the console. You could even create a PR for whatever NUnit runner you would like to use.

Aside from NUnit, I'd be happy to have somebody propose such a feature for the TestCentric GUI runner.

To unsubscribe from this group and stop receiving emails from it, send an email to nunit-discuss+unsubscribe@googlegroups.com.