Generate trx with SpecRun

[carlin scott]

I'd like to deploy my tests to a machine, run them and then publish the results to TFS. I can publish results using MSTest.exe or VSTest.console.exe but they both require a trx results file which SpecRun.exe does not create. I tried using MSTest and VSTest.console to run the tests but they do not recognize any of the specrun tests.

I'm expecting this to be possible because there's a video showing SpecRun tests running on a TFS build server using "Visual Studio Test Runner" which should be the same as vstest.console.exe.

[Gáspár Nagy]

TFS can run tests and collect results from SpecRun without TRX files. The TRX files are kind of obsolete now.

SpecRun has a customizable report template, so it would be possible to generate also TRX files with that, but I don't have a template for that.

Why do you need to publish a test result to TFS through such a file?

Br,
Gaspar

[carlin scott]

I needed the trx file because the only tools I know of for publishing test results to TFS (vstest.console.exe and mstest.exe) require a trx file to do it. I needed to publish the test results outside of the normal build process because the tests cannot from a build agent.

I however figured out how to run the SpecRun tests through vstest.console.exe which generates a trx file or publishes results directly to TFS.

In order to run SpecRun tests with vstest.console.exe I had to reference the following dll's in my test project from the nuget package tools folder: TechTalk.SpecRun.VisualStudio.TestAdapter.dll, TechTalk.SpecRun.Framework.Executor.dll. I also had to enable the "Copy Local" property on these dlls. Finally, I used the following command to execute the tests:

& '"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe"' /TestAdapterPath:.\ /Logger:trx

The logger parameter can also be set to publish the results directly to a TFS build.

[Gáspár Nagy]

Super. Thx for sharing your results.

[Yannick Dextradeur]

Hello,

I'm also interested in this case to generate a trx file from SpecRun and publish the result to TFS.

I tried the procedure you mentionned with the command line and test adapter from specrun. When I run it, it throws an exception when I specify a /TestCaseFilter :

Error: System.ArgumentException: The path is not of a legal form.

   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)

   at System.IO.Path.GetDirectoryName(String path)

   at TechTalk.SpecRun.VisualStudio.TestAdapter.VsTestCaseProvider.GetAbsolutePath(String codeFilePath, String source) in c:\TeamCity\buildAgent\work\1ace6ed01d

0a43bb\VisualStudio\TechTalk.SpecRun.VisualStudio.TestAdapter\VsTestCaseProvider.cs:line 107

   at TechTalk.SpecRun.VisualStudio.TestAdapter.VsTestCaseProvider.CreateVsTestCase(TestItem testItem, String source) in c:\TeamCity\buildAgent\work\1ace6ed01d0

a43bb\VisualStudio\TechTalk.SpecRun.VisualStudio.TestAdapter\VsTestCaseProvider.cs:line 47

   at TechTalk.SpecRun.VisualStudio.TestAdapter.TfsTestFilterProvider.TfsTestItemMatcher(TestItem testItem, String source, TestingContext testingContext) in c:\

TeamCity\buildAgent\work\1ace6ed01d0a43bb\VisualStudio\TechTalk.SpecRun.VisualStudio.TestAdapter\TfsTestFilterProvider.cs:line 0

Or this exception when with no filter:

test thread #0 error: System.ArgumentException: The path is not of a legal form.

   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)

   at System.IO.Path.GetFullPathInternal(String path)

   at System.IO.Path.GetFullPath(String path)

   at TechTalk.SpecRun.Framework.Utils.FileSystemHelper.GetRelativePath(String path, String basePath) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\Utils\FileSystemHelper.cs:line 20

   at TechTalk.SpecRun.Framework.DeploymentSteps.DeploymentContext..ctor(TestProfile testProfile, ITestLogger logger, Int32 testThreadId, IAssemblyReference testAssembly, String testTarget) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\DeploymentSteps\DeploymentContext.cs:line 126

   at TechTalk.SpecRun.Framework.TestThread.InitializeTestThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 148

   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutorFor(TestItem testItem) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 134

   at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 112

   at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\AsyncTestThreadRunner.cs:line 33

test thread error: TechTalk.SpecRun.Framework.SpecRunException: At least one test thread aborted. ---> System.ArgumentException: The path is not of a legal form.

   at System.IO.Path.NormalizePath(String path, Boolean fullCheck, Int32 maxPathLength, Boolean expandShortPaths)

   at System.IO.Path.GetFullPathInternal(String path)

   at System.IO.Path.GetFullPath(String path)

   at TechTalk.SpecRun.Framework.Utils.FileSystemHelper.GetRelativePath(String path, String basePath) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\Utils\FileSystemHelper.cs:line 20

   at TechTalk.SpecRun.Framework.DeploymentSteps.DeploymentContext..ctor(TestProfile testProfile, ITestLogger logger, Int32 testThreadId, IAssemblyReference testAssembly, String testTarget) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\DeploymentSteps\DeploymentContext.cs:line 126

   at TechTalk.SpecRun.Framework.TestThread.InitializeTestThreadExecutor(IAssemblyReference testAssembly, ExecutionModelSettings executionModelSettings, String testTarget) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 148

   at TechTalk.SpecRun.Framework.TestThread.GetThreadExecutorFor(TestItem testItem) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 134

   at TechTalk.SpecRun.Framework.TestThread.Run(ITestExecutionManager executionManagerForRun) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\TestThread.cs:line 112

   at TechTalk.SpecRun.Framework.AsyncTestThreadRunner.RunSync(TestExecutionManager executionManager) in c:\TeamCity\buildAgent\work\1ace6ed01d0a43bb\TechTalk.SpecRun.Framework\AsyncTestThreadRunner.cs:line 33

   --- End of inner exception stack trace ---

Did you run across those problem ? If so, any tip how to resolve them ?

Thanks!

[Yannick Dextradeur]

I found the problem. I had to specify the full path for the DLLs to test.

[Antwan Maddox]

Hi

What video did you find that shows how to run SpecRun on a TFS build server?  Can you share with me that video?

Antwan M

[Andreas Willich]

Hi Antwan

For TFS build server integration with SpecFlow+Runner (aka SpecRun) have a look at the documentation here: http://www.specflow.org/plus/documentation/SpecFlowPlus-and-Build-Servers/

Best reagards

Andreas

--
You received this message because you are subscribed to the Google Groups "SpecFlow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specflow+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.