Advice on how to pass long list of testcases in NUnit Console Runner 3.4.0

[Alex S]

Hi all,

I'm using  NUnit Console and --test=<full_testcase_name> option to run our tests by sending all testcases explicitly via command line, e.g.:

 /C <path_to_nunit_package>/nunit3-console.exe <assembly_file_path> --result=<some_path>/TestResult.xml;format=nunit2 --noheader --workers 1 --test=<testcase_fullname1>,<testcase_fullname2>, ...

Each testcase_fullname1 include "namespace", "class_name" and the actual "test_method_name". All those parameters are quite long, especially the namespace, which is fixed in our case and there is no need to specify it for each test case.

The problem is that when the command-line length hits 8192 or something characters the cmd process fails with "The command line is too long." message.

Is there a different way of passing a long list of testcases avoiding repetition of namespace parameter?

Why do I need to specify all test cases explicitly? I have a list of custom properties like "Customer", "Version", "Module", etc... I run the explore.. Then I parse testcases xml extracting/filtering all the necessary testcase names based on custom properties. Then I pass the final list of test cases to Nunit console.

Is it possible to filter testcases in NUnit Console based on custom properties? I tried using --where option + custom properties but it did not work for me.

Thanks,

Alex

[Alex S]

A good alternative would be "Filtering by Test Id". Test ids is definitely shorter than the full testcase name. Did somebody get the test id filter working?

[Alex S]

Sorry guys, I completely forgot about "--testlist=" option. It perfectly solved my problem!

[Charlie Poole]

Hi Alex,

I was going to mention --testlist= - glad you found it.

Other options for you:

* If your properties have values, you can test for them using
--where "YourPropertyName=SomeValue"
Unfortunately, there is no way to test for the existence of a
property without a value. Maybe we should add that.

* Make your custom properties into custom categories instead and use
--where "cat=YourCustomCategory"

Charlie

2016-09-12 7:23 GMT-07:00 Alex S <yoursh...@gmail.com>:
>
> Sorry guys, I completely forgot about "--testlist=" option. It perfectly
> solved my problem!
>

> --
> You received this message because you are subscribed to the Google Groups
> "NUnit-Discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nunit-discus...@googlegroups.com.
> To post to this group, send email to nunit-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nunit-discuss.
> For more options, visit https://groups.google.com/d/optout.

[Alex S]

Hi Charlie,

Thank you for prompt reply.

I have a complex logic around filtering testcases where I need to exclude/include tests based on multiple properties with values /and on their existence (with no values), so "--where=" option is not enough.

Anyway the --testlist= works perfectly for me.

I'm just curious if filtering by testcase id working at all? I saw it was mentioned in the Wiki: "Filtering by Test Id" but I could not get it working...

Regards,

Alex

[Charlie Poole]

It seems as if the only thing missing for your use case is testing on
property existence. If you file an issue requesting the feature we may
be able to get it in some future release.

Filtering on id should work if you have the correct value for the id.
What have you tried and where did you get the id?

[Alex S]

Hi Charlie,

I got test case ids from explored xml, e.g.:

            <test-suite type="TestFixture" id="16-1006" name="ESBDummyTests3" fullname="TestFramework.Tests.ESB.ESBDummyTests3" classname="TestFramework.Tests.ESB.ESBDummyTests3" runstate="Runnable" testcasecount="4">
              <properties>
                <property name="TestMode" value="backend" />
              </properties>
              <test-case id="16-1007" name="ESB0015_ESBDummyTest15" fullname="TestFramework.Tests.ESB.ESBDummyTests3.ESB0015_ESBDummyTest15" methodname="ESB0015_ESBDummyTest15" classname="TestFramework.Tests.ESB.ESBDummyTests3" runstate="Runnable" seed="1581993905">
                <properties>
                  <property name="Customer" value="~DD" />
                  <property name="Description" value="ESB0015: ESBDummyTest15" />
                </properties>
              </test-case>

For the id filtering I've tried the following so far:

--id=16-1007

--id==16-1007

--id=="16-1007"

Result was in all cases: an error: "Invalid argument: --id==16-1007"

--where "id=16-1007"

--where "id==16-1007"

--where "id==1007"

Result was in all cases: no cases selected, i.e.:

Test Filters
    Where: id==16-1007


Run Settings
    WorkDirectory: C:\_work\...\packages\NUnit.ConsoleRunner.3.4.0\tools
    NumberOfTestWorkers: 1
    ImageRuntimeVersion: 4.0.30319
    ImageTargetFrameworkName: .NETFramework,Version=v4.5.2
    ImageRequiresX86: True
    RunAsX86: True
    ImageRequiresDefaultAppDomainAssemblyResolver: False


Test Run Summary
  Overall result: Passed
  Test Count: 0, Passed: 0, Failed: 0, Inconclusive: 0, Skipped: 0
  Start time: 2016-09-14 07:02:12Z
    End time: 2016-09-14 07:02:12Z
    Duration: 0.058 seconds

What would be the correct usage?

Thanks,

Alex

[Charlie Poole]

First off, let's forget about `--id` - there is no such option on the
command-line.

This form is correct, but won't find anything because there is no id
with that value:
* `--where "id==1007"`

These two are equivalent (`=` and `==` are synonyms):
* `--where "id=16-1007"`
* `--where "id==16-1007"

I would have expected the latter two to work. Since no error is given,
it appears that there is no test with that id in the run. While ids
are only stable for the duration of a run, NUnit should produce the
same ids given the same list of assemblies on the command-line. Are
you using the same assemblies, in the same order, for your --explore
command and your execution command?

[Alex S]

Hi Charlie,

You were right when asking me about --explore execution command. 

It appeared to be for some reason the "--explore" command ran from Visual Studio produced ids with prefix "16" and the command-line execution of the "--explore" produced exactly the same ids with prefix "0".

So, the following examples worked for me:

--where "id=0-1007"
--where "id==0-1007"

Thanks a lot for your clarifications again.

Regards,

Alex

[Charlie Poole]

That's interesting! What exactly do you mean by running the --explore
option from visual studio? I'd like to reproduce this but I'm not sure
what you clicked or entered.

[Alex S]

Hi Charlie,

To reproduce:

1) In Visual Studio GUI open project properties by using menu -> Project -> <project> Properties... 

2) In the <project>.csproj properties file add the following command-line command in "Build Events" -> "Post-build event command line:" window:

 "$(SolutionDir)packages\NUnit.ConsoleRunner.3.4.0\tools\nunit3-console.exe" "$(SolutionDir)<your_solution_name>.sln" --explore=testcases.xml

Alternatively, from text editor add the following line in the <project>.csproj properties file:

 
 <PropertyGroup>
    <PostBuildEvent>"$(SolutionDir)packages\NUnit.ConsoleRunner.3.4.0\tools\nunit3-console.exe" "$(SolutionDir)<your_solution_name>.sln" --explore=testcases.xml</PostBuildEvent>
 </PropertyGroup>

Now, every time you build your project a new testcases.xml file should be generated by "--explore" console option with the latest test cases, where I can observe all ids with the prefix "16".

This PostBuildEvent project setting also works when running project from Jenkins by using "msbuild". So I always have an access to whole up-to-date Test Pack structure (testcases.xml) where I can do more sophisticated test cases filtering before I pass it to my Test Runners.

Regards,

Alex