nunit 3 test results are not recognised as valid test file

613 views
Skip to first unread message

Damian Stanger

unread,
Apr 28, 2016, 11:05:22 AM4/28/16
to go-cd
When i run my tests on the go agent the testresult.xml file is not recognized as valid, the build fails (as it should) when there is a failing test but the tests tab is always empty regardless of success or failure
I'm running Go Version: 16.2.1 and nunit3-console on an azure VM running server 2012 R2
Any pointers would be appreciated.
Thanks Damian

In the Console tab
------------------

14:38:46.340 nunit3-console --where "cat!=Integration" .\foobar.Tests\buildoutput\foobar.Tests.dll
14:38:46.559 NUnit Console Runner 3.2.0
14:38:46.559 Copyright (C) 2016 Charlie Poole
14:38:46.559
14:38:47.515 Runtime Environment
14:38:47.515    OS Version: Microsoft Windows NT 6.3.9600.0
14:38:47.515   CLR Version: 4.0.30319.42000
14:38:47.515
14:38:47.515 Test Files
14:38:47.515     .\foobar.Tests\buildoutput\foobar.Tests.dll
14:38:47.515
14:38:47.528 Test Filters
14:38:47.528     Where: cat!=Integration
14:38:47.528
14:38:51.610
14:38:51.610 Errors and Failures
14:38:51.610
14:38:51.621 1) Failed : foobar.Tests.foobarTests.JourneyIdDropsAnyLeadingZeroFromImei
14:38:51.621 Expected string to start with
14:38:51.621 "0", but
14:38:51.621 "12345/0001/01/01 00:00:00.000" differs near "123" (index 0).
14:38:51.621 at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message) in C:\projects\fluentassertions-vf06b\Src\Shared\Execution\LateBoundTestFramework.cs:line 31
14:38:51.621 at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args) in C:\projects\fluentassertions-vf06b\Src\Core\Execution\AssertionScope.cs:line 197
14:38:51.621 at FluentAssertions.Primitives.StringStartValidator.ValidateAgainstMismatch() in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringStartValidator.cs:line 54
14:38:51.621 at FluentAssertions.Primitives.StringValidator.Validate() in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringValidator.cs:line 45
14:38:51.621 at FluentAssertions.Primitives.StringAssertions.StartWith(String expected, String because, Object[] becauseArgs) in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringAssertions.cs:line 344
14:38:51.621
14:38:51.621 Run Settings
14:38:51.621     WorkDirectory: C:\go\agent5\pipelines\Build-Foobar\TestArtifacts
14:38:51.621     ImageRuntimeVersion: 4.0.30319
14:38:51.621     ImageTargetFrameworkName: .NETFramework,Version=v4.6.1
14:38:51.621     ImageRequiresX86: False
14:38:51.621     ImageRequiresDefaultAppDomainAssemblyResolver: False
14:38:51.621     NumberOfTestWorkers: 2
14:38:51.621
14:38:51.640 Test Run Summary
14:38:51.640   Overall result: Failed
14:38:51.640   Test Count: 3, Passed: 2, Failed: 1, Inconclusive: 0, Skipped: 0
14:38:51.640     Failed Tests - Failures: 1, Errors: 0, Invalid: 0
14:38:51.640   Start time: 2016-04-28 14:38:50Z
14:38:51.640     End time: 2016-04-28 14:38:51Z
14:38:51.640     Duration: 1.291 seconds
14:38:51.640
14:38:51.640 Results (nunit3) saved as TestResult.xml
14:38:51.778 [go] Current job status: failed.

14:38:51.793 [go] Start to create properties Build-Foobar/3/Test/1/Unit on goserver06 [C:\go\agent5]
14:38:51.793 [go] Start to upload Build-Foobar/3/Test/1/Unit on goserver06 [C:\go\agent5]
14:38:51.809 [go] Uploading artifacts from C:\go\agent5\pipelines\Build-Foobar\TestArtifacts\TestResult.xml to [defaultRoot]
14:38:51.856 Ignoring file TestResult.xml - it is not a recognised test file.
14:38:51.997 [go] Uploading artifacts from C:\Users\goadmin\AppData\Local\Temp\cruise-86ec3756-b535-441b-b5a3-79d3a2cb0af6\2fe894db-efa5-494a-8caf-3937e0ca5054\result\index.html to testoutput
14:38:52.028 [go] Uploading artifacts from C:\Users\goadmin\AppData\Local\Temp\cruise-86ec3756-b535-441b-b5a3-79d3a2cb0af6\2fe894db-efa5-494a-8caf-3937e0ca5054\result to testoutput
14:38:52.215 [go] Job completed Build-Foobar/3/Test/1/Unit on goserver06 [C:\go\agent5]


In the Tests tab
----------------

Tests run: 0 , Failures: 0 , Not run: 0 , Time: .000 seconds.

No Tests Run
This project doesn't have any tests


Actual TestResult.xml (success)

-------------------------------
<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="3" result="Passed" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.2.0.0" clr-version="4.0.30319.42000" start-time="2016-04-28 14:47:40Z" end-time="2016-04-28 14:47:40Z" duration="0.179647">
  <command-line><![CDATA["C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe"  --where "cat==Integration " .\foobar.Tests\buildoutput\foobar.Tests.dll]]></command-line>
  <filter>
    <cat>Integration</cat>
  </filter>
  <test-suite type="Assembly" id="0-1004" name="foobar.Tests.dll" fullname="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts\foobar.Tests\buildoutput\foobar.Tests.dll" runstate="Runnable" testcasecount="3" result="Passed" start-time="2016-04-28 14:47:40Z" end-time="2016-04-28 14:47:40Z" duration="0.044840" total="0" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.2.1.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.3.9600.0" platform="Win32NT" cwd="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts" machine-name="GOSERVER" user="goadmin" user-domain="GOSERVER" culture="en-US" uiculture="en-US" os-architecture="x64" />
    <settings>
      <setting name="WorkDirectory" value="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts" />
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageTargetFrameworkName" value=".NETFramework,Version=v4.6.1" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="NumberOfTestWorkers" value="2" />
    </settings>
    <properties>
      <property name="_PID" value="5960" />
      <property name="_APPDOMAIN" value="test-domain-" />
    </properties>
  </test-suite>
</test-run>


Actual TestResult.xml (failure)
-------------------------------

<?xml version="1.0" encoding="utf-8" standalone="no"?>
<test-run id="2" testcasecount="3" result="Failed" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0" engine-version="3.2.0.0" clr-version="4.0.30319.42000" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.290561">
  <command-line><![CDATA["C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe"  --where cat!=Integration .\foobar.Tests\buildoutput\foobar.Tests.dll]]></command-line>
  <filter>
    <not>
      <cat>Integration</cat>
    </not>
  </filter>
  <test-suite type="Assembly" id="0-1004" name="foobar.Tests.dll" fullname="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts\foobar.Tests\buildoutput\foobar.Tests.dll" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.070366" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
    <environment framework-version="3.2.1.0" clr-version="4.0.30319.42000" os-version="Microsoft Windows NT 6.3.9600.0" platform="Win32NT" cwd="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts" machine-name="GOSERVER" user="goadmin" user-domain="GOSERVER" culture="en-US" uiculture="en-US" os-architecture="x64" />
    <settings>
      <setting name="WorkDirectory" value="C:\go\agent5\pipelines\Build-Foobar\TestArtifacts" />
      <setting name="ImageRuntimeVersion" value="4.0.30319" />
      <setting name="ImageTargetFrameworkName" value=".NETFramework,Version=v4.6.1" />
      <setting name="ImageRequiresX86" value="False" />
      <setting name="ImageRequiresDefaultAppDomainAssemblyResolver" value="False" />
      <setting name="NumberOfTestWorkers" value="2" />
    </settings>
    <properties>
      <property name="_PID" value="6100" />
      <property name="_APPDOMAIN" value="test-domain-" />
    </properties>
    <failure>
      <message><![CDATA[One or more child tests had errors]]></message>
    </failure>
    <test-suite type="TestSuite" id="0-1005" name="foobar" fullname="foobar" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031998" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
      <properties />
      <failure>
        <message><![CDATA[One or more child tests had errors]]></message>
      </failure>
      <test-suite type="TestSuite" id="0-1006" name="Tests" fullname="foobar.Tests" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031849" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
        <properties />
        <failure>
          <message><![CDATA[One or more child tests had errors]]></message>
        </failure>
        <test-suite type="TestSuite" id="0-1007" name="aaa" fullname="foobar.Tests.aaa" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031838" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
          <properties />
          <failure>
            <message><![CDATA[One or more child tests had errors]]></message>
          </failure>
          <test-suite type="TestSuite" id="0-1008" name="bbb" fullname="foobar.Tests.aaa.bbb" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031831" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
            <properties />
            <failure>
              <message><![CDATA[One or more child tests had errors]]></message>
            </failure>
            <test-suite type="TestSuite" id="0-1009" name="ccc" fullname="foobar.Tests.aaa.bbb.ccc" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031825" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
              <properties />
              <failure>
                <message><![CDATA[One or more child tests had errors]]></message>
              </failure>
              <test-suite type="TestSuite" id="0-1010" name="Messages" fullname="foobar.Tests.aaa.bbb.ccc.Messages" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031818" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
                <properties />
                <failure>
                  <message><![CDATA[One or more child tests had errors]]></message>
                </failure>
                <test-suite type="TestSuite" id="0-1011" name="Extended" fullname="foobar.Tests.aaa.bbb.ccc.Messages.Extended" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.031789" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
                  <properties />
                  <failure>
                    <message><![CDATA[One or more child tests had errors]]></message>
                  </failure>
                  <test-suite type="TestFixture" id="0-1000" name="bbbDataTests" fullname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests" classname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests" runstate="Runnable" testcasecount="3" result="Failed" site="Child" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="1.022041" total="3" passed="2" failed="1" inconclusive="0" skipped="0" asserts="0">
                    <properties />
                    <failure>
                      <message><![CDATA[One or more child tests had errors]]></message>
                    </failure>
                    <test-case id="0-1001" name="AnEmptybbbDataShouldProduceMinId" fullname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests.AnEmptybbbDataShouldProduceMinId" methodname="AnEmptybbbDataShouldProduceMinId" classname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests" runstate="Runnable" seed="798259755" result="Passed" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:50Z" duration="0.077580" asserts="0">
                      <properties />
                    </test-case>
                    <test-case id="0-1003" name="bbbIdDropsAnyLeadingZeroFromImei" fullname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests.bbbIdDropsAnyLeadingZeroFromImei" methodname="bbbIdDropsAnyLeadingZeroFromImei" classname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests" runstate="Runnable" seed="993609143" result="Failed" start-time="2016-04-28 14:38:50Z" end-time="2016-04-28 14:38:51Z" duration="0.905577" asserts="0">
                      <properties />
                      <failure>
                        <message><![CDATA[Expected string to start with
"0", but
"12345/0001/01/01 00:00:00.000" differs near "123" (index 0).]]></message>
                        <stack-trace><![CDATA[at FluentAssertions.Execution.LateBoundTestFramework.Throw(String message) in C:\projects\fluentassertions-vf06b\Src\Shared\Execution\LateBoundTestFramework.cs:line 31
at FluentAssertions.Execution.AssertionScope.FailWith(String message, Object[] args) in C:\projects\fluentassertions-vf06b\Src\Core\Execution\AssertionScope.cs:line 197
at FluentAssertions.Primitives.StringStartValidator.ValidateAgainstMismatch() in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringStartValidator.cs:line 54
at FluentAssertions.Primitives.StringValidator.Validate() in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringValidator.cs:line 45
at FluentAssertions.Primitives.StringAssertions.StartWith(String expected, String because, Object[] becauseArgs) in C:\projects\fluentassertions-vf06b\Src\Core\Primitives\StringAssertions.cs:line 344
]]></stack-trace>
                      </failure>
                    </test-case>
                    <test-case id="0-1002" name="bbbIdHasImeiAndFormattedStartDate" fullname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests.bbbIdHasImeiAndFormattedStartDate" methodname="bbbIdHasImeiAndFormattedStartDate" classname="foobar.Tests.aaa.bbb.ccc.Messages.Extended.bbbDataTests" runstate="Runnable" seed="1630670862" result="Passed" start-time="2016-04-28 14:38:51Z" end-time="2016-04-28 14:38:51Z" duration="0.000495" asserts="0">
                      <properties />
                    </test-case>
                  </test-suite>
                </test-suite>
              </test-suite>
            </test-suite>
          </test-suite>
        </test-suite>
      </test-suite>
    </test-suite>
  </test-suite>
</test-run>

Zabil C M

unread,
Apr 28, 2016, 12:55:34 PM4/28/16
to go...@googlegroups.com
Hi Damian,

 GoCD supports the Nunit2 report format. There are a lot changes between the Nunit3 style reports and Nunit2 reports.
 If you are using the nunit console runner try passing the "format=nunit2" to make the generated reports compatible.

   

--
You received this message because you are subscribed to the Google Groups "go-cd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to go-cd+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ketan Padegaonkar

unread,
Apr 28, 2016, 3:02:17 PM4/28/16
to go...@googlegroups.com

Also, could you log a bug with the XML snippets you provided here so someone can take a look.

Damian Stanger

unread,
Apr 28, 2016, 4:54:56 PM4/28/16
to go-cd
Thanks this put me on the right track

Fixed :-)
Your reply put me in the right direction, cheers.
The final version of the command Ive got is:

nunit3-console --result:"TestResult.xml;format=nunit2" --where "cat!=Integration" .\foobar.Tests\buildoutput\foobar.Tests.dll

The nunit3 documentation around this area is rubbish but we got there in the end

many thanks
Damian
Reply all
Reply to author
Forward
0 new messages