Test results reports for Cucumber

[Styris]

With my Maven project for Appium (using Cucumber) at the end of running the tests I am looking for a report that lists details such as total no. of TCs, passed/failed/not run etc. However all I am getting is a listing of all the features marked in red or green with some stracktrace for the failed ones. I am pasting a snippet from my POM.xml file here for anyone to suggest what addition/edit I need to make to get the desired results. Please let me know if you have any questions.

Thanks.

[George Dinwiddie]

Styris,

It's a matter of choosing what output formatter you want.
https://www.relishapp.com/cucumber/cucumber/docs/formatters shows a list
of built-in formatters.
https://github.com/cucumber/cucumber/wiki/Custom-Formatters lists some
others that are available. If you can't find a formatter that produces
the report you want, you can write your own plugin to produce it.

- George

--
----------------------------------------------------------------------
* George Dinwiddie * http://blog.gdinwiddie.com
Software Development http://www.idiacomputing.com
Consultant and Coach http://www.agilemaryland.org
----------------------------------------------------------------------

[Ilko Iliev]

I'm using the json output format and I'm getting the overview page correct rendered (features-overview.html):

1) in CucumberTest.java:

@RunWith(Cucumber.class)
@CucumberOptions(plugin = {"pretty", "json:target/cucumber.json"})

2) in pom.xml:

<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>

I've never used the html output directly.

[Kashif BHATTI]

I currently have this in my RunnerClass 

@RunWith(Cucumber.class)

@CucumberOptions

(

plugin = {"pretty", "html:target/cucumber-html-report"},

features = "src/test/resources/",

//tags = ("@inbox, @logoutNewUser"),

glue={"TestCases"},

dryRun = false,

monochrome = true

)

And I sent you the POM file already. Where exactly do you think I can make changes ?

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/7QllDj3Y4sw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ilko Iliev]

2) in pom.xml:

<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>

On Thursday, February 25, 2016 at 3:12:48 AM UTC+1, Styris wrote:

[Ilko Iliev]

if it still does not work try to update your plugin, I'm using:

version: cucumber-reporting-0.2.1

[Kashif BHATTI]

Not sure what I'm missing but I get an html report that just has features listed in red and green...and a cucumber.son file that is all son format. Nothing in the report where I see total TCs, total passed/failed sort of information. Once I again I am sending my pom and cucumber runner class :/.

Could it be that I'm commenting out the goals tag? Could it be that am missing some JAR ? or maven dependency ?

Thanks.

--

[Aslak Hellesøy]

On Friday, March 11, 2016 at 1:43:21 AM UTC, Styris wrote:

Not sure what I'm missing but I get an html report that just has features listed in red and green...

That sounds correct. Were you expecting something different?

 

and a cucumber.son file that is all son format.

I think you mean JSON format?

 

Nothing in the report where I see total TCs, total passed/failed sort of information.

The report currently doesn't display this information.

In fact, the HTML reporting of Cucumber is something we're not very proud of. It differs a lot between implementations, doesn't look great and lacks a lot of the information you're describing.

The core teams of all the Cucumber implementations is meeting up next week, and one of the things we're likely to focus on is improved reporting.

Cheers,

Aslak

 

Once I again I am sending my pom and cucumber runner class :/.

Could it be that I'm commenting out the goals tag? Could it be that am missing some JAR ? or maven dependency ?

Sounds like you're doing everything correctly.

 

Thanks.

On Tue, Mar 8, 2016 at 12:58 AM, Ilko Iliev <ilkom...@gmail.com> wrote:

if it still does not work try to update your plugin, I'm using:

version: cucumber-reporting-0.2.1

On Tuesday, March 8, 2016 at 9:56:43 AM UTC+1, Ilko Iliev wrote:

2) in pom.xml:

<cucumberOutput>${project.build.directory}/cucumber.json</cucumberOutput>


On Thursday, February 25, 2016 at 3:12:48 AM UTC+1, Styris wrote:

With my Maven project for Appium (using Cucumber) at the end of running the tests I am looking for a report that lists details such as total no. of TCs, passed/failed/not run etc. However all I am getting is a listing of all the features marked in red or green with some stracktrace for the failed ones. I am pasting a snippet from my POM.xml file here for anyone to suggest what addition/edit I need to make to get the desired results. Please let me know if you have any questions.

Thanks.

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/7QllDj3Y4sw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

The Cucumber logo is the intellectual property of Cucumber Ltd, a limited company registered in Scotland, number 456793.

UK Headquarters: Cucumber Ltd, Drumsyniebeg, Lochgoilhead, Cairndow, Argyll, PA24 8AN UK.

CONFIDENTIALITY NOTICE: The information in this e-mail is confidential and privileged; it is intended for use solely by the individual or entity named as the recipient hereof. Disclosure, copying, distribution, or use of the contents of this e-mail by persons other than the intended recipient is strictly prohibited and may violate applicable laws. If you have received this e-mail in error, please delete the original message and notify us by email immediately. Thank you. Cucumber Ltd.

[Kashif BHATTI]

Well, right now what I see generated is two reports...one is an HTML report right...and all it does is just display all the features with results, either in red or green. How can my Mgmt. tell what were the total no. of TCs, how many passed and failed? The Cucumber.json file that is generated is not even in a web-viewable format...all it is..is in JSON format so how does that help from a reporting perspective?

The reason I though that the HTML report will be more detailed is because I came across this website: http://damienfremont.com/2015/07/30/how-to-cucumber-test-reporting-plugin-with-maven-and-java/

Notice on that site there is a snapshot for Cucumber-Reports HTML Plugin...notice how they are getting all the detailed information in all that? I thought that would be doable for anyone using Cucumber?

Thans Aslak.

[Matthew]

On Saturday, March 12, 2016 at 6:46:53 AM UTC+13, Styris wrote:

Well, right now what I see generated is two reports...one is an HTML report right...and all it does is just display all the features with results, either in red or green. How can my Mgmt. tell what were the total no. of TCs, how many passed and failed? The Cucumber.json file that is generated is not even in a web-viewable format...all it is..is in JSON format so how does that help from a reporting perspective?

The reason I though that the HTML report will be more detailed is because I came across this website: http://damienfremont.com/2015/07/30/how-to-cucumber-test-reporting-plugin-with-maven-and-java/

This example uses the masterthought plugin to take the json report and then process it into the html report shown - you could give that a go, and see if it gave you hat you wanted?

Cheers,

Matt

[Kashif Bhatti]

Hmm..I thought I was doing that already, no? At least my files would say so..unless am missing some jar etc.

Thanks

Sent from my iPhone

[Björn Rasmusson]

Styris wrote:

Hmm..I thought I was doing that already, no? At least my files would say so..unless am missing some jar etc.

Thanks

Sent from my iPhone

On Mar 12, 2016, at 10:09, Matthew <matthew....@equinox.co.nz> wrote:

On Saturday, March 12, 2016 at 6:46:53 AM UTC+13, Styris wrote:

Well, right now what I see generated is two reports...one is an HTML report right...and all it does is just display all the features with results, either in red or green. How can my Mgmt. tell what were the total no. of TCs, how many passed and failed? The Cucumber.json file that is generated is not even in a web-viewable format...all it is..is in JSON format so how does that help from a reporting perspective?

The reason I though that the HTML report will be more detailed is because I came across this website: http://damienfremont.com/2015/07/30/how-to-cucumber-test-reporting-plugin-with-maven-and-java/

You need to raise you level of attention to detail when you copy examples you find on the net.
If you look closely in the "Cucumber-Reports HTML Plugin" section in the blog post you refer to, you will see the different directories are specified in @CucumberOptions:

        "html:target/site/cucumber-pretty",

and in the pom.xml:

        <outputDirectory>${project.build.directory}/site/cucumber-reports</outputDirectory>

But you specify the same:

        "html:target/cucumber-html-report",

and:

        <outputDirectory>target/cucumber-html-report</outputDirectory>

That means that you let two different tools generate their html report files in the same directory, not something that is likely to work without problem.
There are also other differences between the example in the blog post and yours. If you do exactly as in the blog you will get the html-report you expect.
After you have a working example that is exactly as in the blog post, then you can to whatever changes you want. If you then make a change and it stops working, you can just revert the latest change and be back with a working solution again.

One change that you would want make, after you checked in target/site/cucumber-pretty/ and see that you are not interested in the html-report there (generated by what is called "Native HTML Plugin" in the blog post), is to remove

        "html:target/site/cucumber-pretty",

from @CucumberOptions.

Regards
Björn
 

[Kashif BHATTI]

I think you did not get my point. On that website ( http://damienfremont.com/2015/07/30/how-to-cucumber-test-reporting-plugin-with-maven-and-java/) there is a section titled  "Cucumber-Reports HTML Plugin". Under that there are 2 screenshots that have quite a bit of detail like no. of TCs passed/failed. 

What you explained to me is going to get me a Native HTML report...but just a result of all the features that were run...and marked as green or red. There is no way in the reports for me, currently, to tell the total no. of TCs..passed/failed etc.

Pretty much tried everything...not even sure if there is anything else left to try. Maybe it is just something Cucumber will not support.

Thanks.

[Ilko Iliev]

Could you try with a sample project with only the simplest configuration - json output in the test-runner and <cucumberOutput> pointing to the ABSOLUTE path (as I posted it above) of the generated json file?
As I wrote I haven't tried to mix the "native" cucumber html reporting and the mastertought reporting plugin. If you configured everything ok you should find in your target/site/cucumber-reports folder file called features-overview.html, which should contain the information from the screenshot, you are referring to.
I don't believe that both plugins interfere somehow, but who knows - may be the correct results are overwritten from the html plugin.

I'm getting these metrics and the only difference I can spot is that I'm using the absolute path to the cucumber.json and you are using the relative - perhaps the json to html converter can't find your cucumber.json file? The example you are using also uses the ABSOLUTE PATH - perhaps that is the culprit.

[Kashif BHATTI]

If you don't mind can you please send me or paste here, your runner class and pom.xml file (you can send it to just me if you'd like) ? I would like to see what it could be that I may be missing.

Thanks.

[Ilko Iliev]

https://github.com/ilkomiliev/bdd-cucumber-simple

is there any chance that you run maven only to the test or package phase? You must call the verify phase in maven to activate the plugin, as it is configured at the moment:

mvn clean verify

you should get at the end of the maven console something like this:

[INFO] --- maven-cucumber-reporting:0.1.0:generate (reporting) @ bdd-cucumber-simple ---
Downloading: http://repo.maven.apache.org/maven2/com/google/guava/guava/r09/guava-r09.jar
Downloaded: http://repo.maven.apache.org/maven2/com/google/guava/guava/r09/guava-r09.jar (1118 KB at 201.8 KB/sec)
[INFO] About to generate Cucumber report.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11.198 s
[INFO] Finished at: 2016-03-16T09:43:22+01:00
[INFO] Final Memory: 22M/213M
[INFO] ------------------------------------------------------------------------

[Ilko Iliev]

I've used code excerpts from the cucumber demos - if I'm violating some license issues please let me know to bring the sample down.

[Kashif BHATTI]

OK I ran the mvn clean verify command and the error I got was:

[INFO] Building iOSSmokeTest 0.0.1-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[WARNING] The POM for net.masterthought:maven-cucumber-reporting:jar:0.2.1 is missing, no dependency information available

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 0.251 s

[INFO] Finished at: 2016-03-16T18:51:19-07:00

[INFO] Final Memory: 6M/123M

[INFO] ------------------------------------------------------------------------

[ERROR] Plugin net.masterthought:maven-cucumber-reporting:0.2.1 or one of its dependencies could not be resolved: Failure to find net.masterthought:maven-cucumber-reporting:jar:0.2.1 in http://repo.bodar.com was cached in the local repository, resolution will not be reattempted until the update interval of repo.bodar.com has elapsed or updates are forced -> [Help 1]

Can you tell me where and what am missing? Please note that in addition to all that in my Project there is also a project called 'lib' under the target folder. The 'lib' folder contains totallylazy-1.81 JARs. Hopefully we are getting close to a resolution here :-(

[Matthew]

On Thursday, March 17, 2016 at 3:02:59 PM UTC+13, Styris wrote:

OK I ran the mvn clean verify command and the error I got was:

[INFO] Building iOSSmokeTest 0.0.1-SNAPSHOT

[INFO] ------------------------------------------------------------------------

[WARNING] The POM for net.masterthought:maven-cucumber-reporting:jar:0.2.1 is missing, no dependency information available

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 0.251 s

[INFO] Finished at: 2016-03-16T18:51:19-07:00

[INFO] Final Memory: 6M/123M

[INFO] ------------------------------------------------------------------------

[ERROR] Plugin net.masterthought:maven-cucumber-reporting:0.2.1 or one of its dependencies could not be resolved: Failure to find net.masterthought:maven-cucumber-reporting:jar:0.2.1 in http://repo.bodar.com was cached in the local repository, resolution will not be reattempted until the update interval of repo.bodar.com has elapsed or updates are forced -> [Help 1]

So this is not resolving the masterthought reporting jar from your local repo (it may just be the pom is missing) - you'll need to get it resolving (although should be mirrored from Maven Central - but it seems the setup you have has issues getting everything), so that the build can run.

Cheers,

Matt

 

[Ilko Iliev]

correct - I'd delete the corresponding folders under .m2 to force the update - the error means no reattempt will be made until the update timeout as set has elapsed. And I'm always have issues forcing maven to update after such error - the most reliable way is to delete the folders / artefacts causing problem.