Publishing vstest results?

[Jeff Dege]

I have a VS2013 project that I'm building with Jenkins. Now I'm trying to get running the unit tests to be a part of the build process.

I'm running the tests with the VSTest Runner plugin, and that's working fine. But I've not figured out how to publish the results.

Currently, I'm trying the MSTest plugin, but it's only sorta working.

I have the VSTest Runner configured to log results to a trx file, and the file shows up in the TestResults directory of the Jenkins workspace, with a complicated, unpredictable name.

And I have the MSTest plugin configured to look for TestResults\*.trx - which I'm hoping will work for locating the file that the runner generated.

When I look at a build's Test Result, I a red/blue failure/success bar (8 failures, 66 tests). And I see two grids below. The first is labeled "All Failed Tests", and has a row for each failed test. The second is labeled "All Tests", and has only one row, with a package of "(root)".

When I try to expand any of the failed tests, (clicking on the blue '+'), the row expands to show a pane displaying "HTTP ERROR 404".

So, the questions:

1. Does the MSTest plugin support vstest.console.exe's trx output?
2. If so, why am I only getting partial results? 
3. Is it because I'm not passing the filename correctly? 
4. If I'm not, how should I pass the filename between the two plugins?
5. And if the MSTest plugin does not support vstest.console.exe's trx output, what do I do?
6. Is there some other plugin I should be using? Some other tool?

[Glenn V]

I think I ran into similar problems a while ago.

These open jenkins issues are related:

https://issues.jenkins-ci.org/browse/JENKINS-21424

https://issues.jenkins-ci.org/browse/JENKINS-19360

Attached to Jenkins-19360 is an xsl file that you can use to transform the trx file that vstest produces to an xml file that the "Publish Junit test result report" post build action can handle.

Kind regards,

Glenn

[Jeff Dege]

Any hints as o how to use the xsl file to transform the trx?

Is there a hook within the vstest runner or mstest plugins, where I can plug in the xsl?

Or do I need to add a new step to do the conversion?  Using what tool?

[Jeff Dege]

OK, I have added a Windows Batch Command task, that's using Saxonica's XLST transform to the VsTest .trx file to what is supposed to be an MsTest .trx file that the Jenkins plug-in can handle, using the .xsl file included in Jenkins-19360.

It's not working at all.

Where the untransformed file would display the passed and failed tests, but broke when you tried to display the results of a test, the plugin parses the transformed file and reports no tests at all.

I can't have been the first to deal with this.  Is there someplace else on the net I should be asking for help?

[Glenn V]

The xslt does not transform to MsTest format, but to JUnit format. You can import the resulting xml file with the "publish junit test result report" post build task.

Kind regards,

Glenn

[Rob D]

Jeff, did you get this working. I just ran into the same issue and discovered your thread. I am very new to Jenkins so if you or anyone else can explain how to display the results of the vstestrunner plugin so that a beginner can understand it, I would be extremely grateful.

Rob

[Sebok, Marton]

Hi!

 

I’ve found two incompatibilities between the publish functionality of the mstest plugin and vstest.console generated trx files:

1.       Vstest.console generates the name of the trx file dynamically

2.       Mstest plugin tries to parse fully qualified type names in the trx, where vstest.console writes simple names.

 

I ‘ve wrote a simple powershell script to translate the test result:

 

# Convert vstest.console generated trx files to be compatible with jenkins mstest extension

# (needs a comma in className)

 

if(Test-Path "${env:WORKSPACE}\TestResults\")

{

  $trxPath = Get-Item "${env:WORKSPACE}\TestResults\*.trx" | Select-Object -Last 1;

 

  [System.Xml.XmlDocument] $xmlDoc = new-object System.Xml.XmlDocument

  $xmlDoc.Load($trxPath)

 

  $nsmgr = New-Object System.Xml.XmlNamespaceManager $xmlDoc.NameTable

  $nsmgr.AddNamespace("t", "http://microsoft.com/schemas/VisualStudio/TeamTest/2010")

 

  $testMethodList = $xmlDoc.SelectNodes("//t:TestMethod[not(contains(@className, ','))]", $nsmgr)

 

  foreach($testMethod in $testMethodList)

  {

    ($testMethod.Attributes.ItemOf("className")).Value += ","

  }

 

  $xmlDoc.Save("${env:WORKSPACE}\testresult.trx")

}

 

After running this this script the mstest plugin can be configured to search “testresult.trx”

 

I hope I could help.

 

Marci

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Dege]

First - added a build action: run unit tests with VSTest.console

Second - added a build-action: Execute Windows batch command to convert the results from vstest to junit

Third - added a post-build action: Publish JUnit test result report

What VSTest.console outputs is XML, what the JUnit report publisher expects is XML. But the schemas differ, and the filenames are different.

The mstest-to-junit.xsl file, attached to https://issues.jenkins-ci.org/browse/JENKINS-19360 is an XSL transform,which will convert the an XML file from the VSTest.console schema to the Junit schema. But how to run it from Jenkins?

I installed Saxonica's xslt tools on the build machine: http://www.saxonica.com/welcome/welcome.xml

I created a directory in Jenkins_home for my own custom stuff, and in that directory I placed the mstest-to-junit.xsl file, and a batch file:

@ECHO OFF

SET "WORKINGDIR=%~1"

FOR %%t IN ("%WORKINGDIR%\*.trx") DO (

    "d:\Program Files\Saxonica\SaxonHE9.5N\bin\Transform.exe" "-s:%%t" "-xsl:%JENKINS_HOME%\KorTerraStuff\mstest-junit.xsl" -versionmsg:off "-o:%WORKINGDIR%\testresults.xml"

)

DEL "%WORKINGDIR%\*.trx"

Part of the problem is that VSTest.console outputs into a time-stamped filename. The batchfile is wildcarded, and will convert any .trx file into "testresults.xml".

The batch command, in Jenkins:

ECHO Convert VSTest.console output to JUnit format

CD %WORKSPACE%

%JENKINS_HOME%\MyStuff\ConvertVsTest2Junit.bat %WORKSPACE%\TestResults

[Rob D]

Thanks Marton and Jeff. I was out for a few days and just now had a chance to work on this. Test results are now displaying.

Rob

[James Telfer]

For those that are reading this thread and are interested, there is a plugin for this: Jenkins MSTest plugin

It wraps the transformation to JUnit format and publishes the result, making it a lot easier to setup. You pass the TRX in directly.

There's also a MSTest runner plugin if you're interested too that works in concert with the MSTest plugin.

HTH,

JT

[ok999]

@James any idea on how to use the plugin via the new jenkins pipeline. I am using the MS Test plugin to publish the .trx. 

But after adopting the new pipeline, i am not sure on how to call this action via a workflow.