Test Results As XML

[xDazedx]

I am looking to integrate our test results into another app and was wondering if there is a way to create a separate output of the test results to XML?. I still want the html report!  I saw that there was SpecRunToXML in nuget but the date is 2012 so I am doubting that will work with current specflow/specrun.

Thanks!

[Andreas Willich]

Hi

We added with version 1.5 of the SpecFlow+Runner the option to generate multiple reports for a single test run.

Have a look at http://www.specflow.org/plus/documentation/SpecFlowPlus-Runner-Profiles/#Report

Best regards

Andreas

On Tue, Jul 19, 2016 at 8:31 PM xDazedx <tcjco...@gmail.com> wrote:

I am looking to integrate our test results into another app and was wondering if there is a way to create a separate output of the test results to XML?. I still want the html report!  I saw that there was SpecRunToXML in nuget but the date is 2012 so I am doubting that will work with current specflow/specrun.

Thanks!

--
You received this message because you are subscribed to the Google Groups "SpecRun" group.
To unsubscribe from this group and stop receiving emails from it, send an email to specrun+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Stephen McCafferty]

Generating XML output is something we have discussed for a future release. We have received several requests for the ability to process the report output in a variety of ways: historical reports, post-processing, merging reports etc. XML is better suited to many of these use cases.

Would you mind letting us know what you want to do with the XML?

[xDazedx]

Thanks guys, The xml will be parsed and eventually loaded into Tableau. What is the ETA on the new reporting formats? 

[Stephen McCafferty]

Thanks guys, The xml will be parsed and eventually loaded into Tableau. What is the ETA on the new reporting formats? 

That's hard to say, given that it's holiday season right now. I think we will have a clearer picture once everyone is back and we discuss the next steps, and of course estimate the actual effort involved. I don't want to make promises we can't keep, so no ETA as of now :)

However, seeing as outputting XML would be the first step towards meeting other feature requests for more flexible reports (merging, historical etc.), I would expect this to be something we look at implementing sooner rather than later. There definitely seems to be a demand for more flexible reports in general, judging by the number of threads on this subject recently.

[xDazedx]

Stephen - Just wanted to check and see if there any updates on this? :)

[Stephen McCafferty]

Andreas plans to focus on the implementation of both an XML and JSON export next week.

Assuming no calamities befall us in the meantime, I would expect us to have some results in the coming weeks.

[Todd Johnson]

You guys rock!

On Fri, Sep 9, 2016 at 7:09 AM, Stephen McCafferty <stephen.m...@techtalk.at> wrote:

Andreas plans to focus on the implementation of both an XML and JSON export next week.

Assuming no calamities befall us in the meantime, I would expect us to have some results in the coming weeks.

--

You received this message because you are subscribed to the Google Groups "SpecRun" group.

To unsubscribe from this group and stop receiving emails from it, send an email to specrun+unsubscribe@googlegroups.com.

[Stephen McCafferty]

We have published a pre-release version of SpecFlow (1.6.0-rc001) that includes two additional report templates:

• ReportTemplate_Json.cshtml
• ReportTemplate_Xml.cshtml

These files are located in the templates directory (where the standard report template, ReportTemplate.cshtml, is also located). I hope the names are self explanatory; one template outputs the data in JSON format, the other outputs XML.

As this is a pre-release package, you will need to enable the "Include prerelease" option in Visual Studio's NuGet Package Manager to see them in the list.

It would be great if you could try the templates out and let us know if there are any issues.

[Todd Johnson]

Thanks Stephen! I will add these to a project today.

[xDazedx]

 

Stephen - Ran into a slight glitch. I was able to grab the RC's ok but I can't get v2.2.0-preview20161020 from nuget. Output shows "Attempting to gather dependency information for multiple packages with respect to project Tests, targeting .NETFramework, version=v4.5" and then it just stops. If I go without it, I do not see tests in test explorer. I am using VS2015.  Since I have you here, will I be able to output the report in HTML and XML or is this one or the other? Hoping for both! 

[Stephen McCafferty]

It should suffice to update to SpecRun.SpecFlow 1.6.0-rc001. SpecFlow 2.2.0 is not required for the report templates, which is why it isn't a dependency of SpecFlow+ 1.6.

You can use version 2.2.0 of SpecFlow if you want, but you will need to install the SpecRun.SpecFlow 2.2.0 package, which will install SpecFlow 2.2.0 and SpecFlow+ 1.6.0.

Maybe you can try again and see if you still have issues with the tests not displaying.

-----

The template itself only outputs XML. But you can generate any number of reports for the same test run. We recently added the <Report> element to SpecRun profiles, and you can specify additional templates here (in addition to the one specified in the <Settings> element). You can find some more info here. So to get both HTML and XML just reference the appropriate templates here.

[xDazedx]

Stephen - I have tests now but am stuck at one thing. I am using the below and I am throwing errors using  copyAlsoToBaseFolder. If I mouseover it it says: The "copyAlsoToBaseFolder" attribute is not declared. 

Building the proj throws:

------ Discover test started ------

An exception occurred while test discoverer 'SpecRunTestDiscoverer' was loading tests. Exception: Invalid configuration file: There is an error in XML document (6, 67). -> Unknown configuration node: copyAlsoToBaseFolder

========== Discover test finished: 0 found (0:00:00.1830183) ==========

<?xml version="1.0" encoding="utf-8"?> <TestProfile xmlns="http://www.specflow.org/schemas/plus/TestProfile/1.5"> <Settings projectName="Tests" projectId="{bf607512-78cf-41f7-b7b6-2b2a524ce3c9}" name="Multiple Reports"/> <Execution stopAfterFailures="100" testThreadCount="1" testSchedulingMode="Sequential" /> <Report> <Template name="ReportTemplate.cshtml" copyAlsoToBaseFolder ="true" outputName="Report1.html" /> <Template name="ReportTemplate_Xml.cshtml" copyAlsoToBaseFolder="true" outputName="Report2.xml" /> </Report>

I ran it without that call and had this is my output..

An exception occurred while invoking executor 'executor://specrun/executorV1.6.0': Object reference not set to an instance of an object.

========== Run test finished: 29 run (0:02:21.843363) ==========

[Stephen McCafferty]

I had a similar issue to you. I then went and looked at the source code, and it seems that the copyAlsoToBaseFolder option belongs in the <Report> element, and not in the <Template> element.

I used the following to test, and it seemed to work:
  <Report copyAlsoToBaseFolder="true">
    <Template name ="ReportTemplate.cshtml" outputName="extra_report.html" ></Template>   
  </Report>

It doesn't seem to work if copyAlsoToBaseFolder is in the <Template> element.

I need to check with Andreas if this is the intended behaviour (in which case I need to fix the documentation). Andreas is on holiday until the middle of next week, so I won't know more until then.

In the meantime, does that fix your issue?

[xDazedx]

That helped.  I was throwing an error that the ReportTemplate.cshtml did not exist so it might be good to add to the docs that the templates must be added to the proj output folder. I set mine to copy always. That worked on the first run. The second run threw the below. So it errors if the report already exists vs overlaying it. The xml report did not appear so I deleted the Report1.html and reran. That produced the two reports but I threw another error shown at the bottom. 

  <Settings projectName="Tests" projectId="{bf607512-78cf-41f7-b7b6-2b2a524ce3c9}" name="Multiple Reports"/>

  <Execution stopAfterFailures="100" testThreadCount="6" testSchedulingMode="Sequential" />
  <Report copyAlsoToBaseFolder="true">
    <Template name="ReportTemplate.cshtml" outputName="Report1.html" />
    <Template name="ReportTemplate_Xml.cshtml" outputName="Report2.xml" />
  </Report>

[ERROR] Timed out after 10 seconds
System.IO.IOException: The file 'D:\SM_Test_SpecFlowChanges\Selenium_Cms_Automation\Tests\bin\Debug\Report1.html' already exists.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at TechTalk.SpecRun.Framework.Reporting.ExecutionReporter.ReportResult(TestRunResult result)
   at TechTalk.SpecRun.Framework.ExecutionEngine.ExecuteTestSuite(TestProfile testProfile, TestRunExecutionConfiguration executionConfiguration, IExecutionContainerBuilder containerBuilder)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestExecutor.<>c__DisplayClass9_0.<ExecuteTestSuite>b__1()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
   at TechTalk.SpecRun.Framework.Reporting.ExecutionReporter.ReportResult(TestRunResult result)
   at TechTalk.SpecRun.Framework.ExecutionEngine.ExecuteTestSuite(TestProfile testProfile, TestRunExecutionConfiguration executionConfiguration, IExecutionContainerBuilder containerBuilder)
   at TechTalk.SpecRun.VisualStudio.TestAdapter.SpecRunTestExecutor.<>c__DisplayClass9_0.<ExecuteTestSuite>b__1()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()
Log file: file:///D:\SM_Test_SpecFlowChanges\Selenium_Cms_Automation\TestResults\Tests_Multiple%20Reports_2016-10-27T090055.log

An exception occurred while invoking executor 'executor://specrun/executorV1.6.0': Object reference not set to an instance of an object.

========== Run test finished: 31 run (0:02:49.5629546) ==========

Execution Time: 00:02:06.8146634
Report file: file:///D:\SM_Test_SpecFlowChanges\Selenium_Cms_Automation\TestResults\Report1.html
Report file: file:///D:\SM_Test_SpecFlowChanges\Selenium_Cms_Automation\TestResults\Report2.xml
Log file: file:///D:\SM_Test_SpecFlowChanges\Selenium_Cms_Automation\TestResults\Tests_Multiple%20Reports_2016-10-27T090800.log
An exception occurred while invoking executor 'executor://specrun/executorV1.6.0': An item with the same key has already been added.
========== Run test finished: 29 run (0:02:20.2057682) ==========

[xDazedx]

Stephen - How do I add a time:date:stamp on the report output so that there is no error for duplicates?

[Stephen McCafferty]

How would you expect SpecFlow+ to handle cases where a report with the same name already exists? Would you expect the existing report to be overwritten if the new one has the same name? We definitely should have a better solution than throwing an error...

I don't think you currently can use the available placeholders (which include environmental variables like %DATE%) in the name of the reports, but I do think that there needs to be some mechanism to handle this case. I will talk to Andreas about this when he is back next week.

[xDazedx]

Thanks Stephen. I think I would automatically append a timestamp so you would never duplicate. This is kind of mentioned on this link but apparently is not implemented here. 

 time stamp (date in YYYY-MM-DD format and time in HHMMSS format) is appended to the generated report. The final name of the generated report is composed as follows: <projectName>_<name>_YYYY-MM-DDTHHMMSS




[xDazedx]

As an alternative you could try the Microsoft approach and jif a duplicate is found like Report1.html they would create Report1(1).html. The timestamp would be better though.

[xDazedx]

Any updates in regards to appending a time stamp? 

Thanks!

[Stephen McCafferty]

I brought up the issue of time stamps and overwriting existing files with Andreas, so he is aware of the problem. I'm not sure when we'll publish a new package as we're still gathering feedback on the new features to see what needs to be improved. A time stamp is on the To-Do list, we are not 100% sure how best to deal with existing templates with the same name, but agree that this case needs to be handled better.

I've also since reviewed the documentation, and I noticed that the copyAlsoToBaseFolder is correctly documented as belogning to the <Report> element, not the <Template> element. We discussed the current setup and weren't sure whether it should actually be a report-specific option or not. Do you have a use case where you would want to set the option individually for each report?

[Todd Johnson]

Thanks Stephen. For Jenkins this is not an issue because I rename the report anyway. It is just an issue if you are running locally. As for the copyAlsoToBaseFolder I can't think of a reason for it to be individually for each report. It is working fine for me. 

On Fri, Nov 11, 2016 at 5:55 AM, Stephen McCafferty <stephen.m...@techtalk.at> wrote:

I brought up the issue of time stamps and overwriting existing files with Andreas, so he is aware of the problem. I'm not sure when we'll publish a new package as we're still gathering feedback on the new features to see what needs to be improved. A time stamp is on the To-Do list, we are not 100% sure how best to deal with existing templates with the same name, but agree that this case needs to be handled better.

I've also since reviewed the documentation, and I noticed that the copyAlsoToBaseFolder is correctly documented as belogning to the <Report> element, not the <Template> element. We discussed the current setup and weren't sure whether it should actually be a report-specific option or not. Do you have a use case where you would want to set the option individually for each report?

--

[Dibyaranjan Kar]

Hi,

I was trying to have tags as part xml output but it seems it is not part of the template. Did anyone try that out?

Thanks
Dibyaranjan

[Stephen McCafferty]

You can extend the XML template with something like the following:

<Tags>

                        @foreach (var tag in test.TestNode.Tags)

                        {

                            <Tag name="@tag"/>

                        }

</Tags>

[prsd.s...@gmail.com]

Thanks Stephen.!!!. We are able to generate XML Report with SpecRun pre-release i.e. 1.6.0-rc002.

Could you please let us know, when are you planning to push this to market. as its pre-release version we can't directly use it in our project.

Thanks,

Varaprasad

[Stephen McCafferty]

There are still a couple of bugs that need squashing in the RC, and we are in the process of working on SpecFlow as a Sim-Ship partner for VS 2017, so I think realistically we are looking at at least a month or so from now until the release is made official.

[Stephen McCafferty]

SpecFlow+ 1.6.0 has now been officially released. We ended up squashing a lot more minor bugs than expected :)

We have extended the <Report> element with an additional attribute, existingFileHandlingStrategy to deal with the issues of existing report files being automatically overwritten if the new report has the same name.

This attribute determines the behaviour if an existing file with the same name already exists. Possible values:
Overwrite (default): Overwrites existing files with the same name
IncrementFilename:Follows the Windows paradigm of adding an incremental suffix (e.g. "(1)", "(2)" etc.) to the end of file names. This will retain the existing file with its original name and result in a new file being generated with the appropriate incremental suffix.

[Alexander Bakanov]

Hi,

I'm also looking to integrate specrun test results with alternative reporting system which expects set of json files.

I was trying to use json cshtml template and then deserialize results using report json file, but i've got a lot of issues with json deserializer (abstract classes, no default constructors, etc.).

Are there any ways to access Test Results directly without razor template?

Or maybe you can suggest other way to solve my problem.

Thanks in advance.