Pending scenaios are reported as passed

[Ali Ostadsaraie]

Hi All,

I've just noticed pending scenarios (skipped ones) are reported as passed in Thucydides report.

This is confusing when you run 100 test cases, which are passed, out of 200 and reports shows 200 test cases are passed. Please look at the attached screenshot.

This happens after updating from 0.9.229. When I roll back to 0.9.229 report looks fine.

Just wondering, if anybody else has experienced this issue.

Thanks.

Ali

[Ali Ostadsaraie]

The new attached screenshot shows the test report of the same scenarios using Thucydides 0.9.229. This reports looks fine as the skipped scenario is reported as "Pending" not "Passed".

Due to this issue I cannot update to any version above 0.9.229.

[Ali Ostadsaraie]

Any thought/reply?

[Ali Ostadsaraie]

Could somebody please let me know, if thucydides report works properly for you? As it displays all pending/skipped scenarios "passed" for me.

[John Smart]

I haven't been able to reproduce this - can you create an example project starting from the archetype that reproduces the issue?

--
You received this message because you are subscribed to the Google Groups "Thucydides Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

___________________________________________________
John Smart | Wakaleo Consulting  |  +61 407 247 642
Optimizing your software development process
http://www.wakaleo.com  |  john....@wakaleo.com
___________________________________________________

The dates for the 2014 BDD workshops have been scheduled! Check out our upcoming BDD/TDD Master classes and our Advanced BDD Requirements Workshops, coming soon to Sydney and Melbourne!

___________________________________________________

[Aleksey Gruba]

Hi,

I suppose that the problem is related to the tests that are marked as pending programmatically (the tests marked with @Pending annotation are processed correctly).

I have the same problem - assume methods are widely used in my test suites and starting from 0.9.244 versions all tests with failed assume conditions are reported as passed (instead of pending status as it was before Thucydides 0.9.244). Simple project generated from net.thucydides:thucydides-simple-archetype archetype can be found here: https://github.com/ovenal/demo-thucydides-pending-test
The easiest way to demonstrate this problem is to add a test like this (in fact the sample project just contains this method):

    @Test
    public void demonstrate_that_pending_test_is_marked_as_passed() {
        Assume.assumeTrue(false);
    }

Also in attachment you can find a screenshot with an illustration the difference between 0.9.229 and 0.9.244 versions.

[Ali Ostadsaraie]

I don't have such a tag in my test cases, however, I use @skip in some test scenarios. The question is that why this issue doesn't show up in 0.9.229?

Cheers,

Ali

[John Smart]

Thanks, I've reproduced this - should be fixed in the next release.

[Andrey Tolpeev]

Hello, John.

Was this issue fixed in 0.9.250? I can still reproduce it on the latest version.

In my case I ran test scenarios using meta filter, and all scenarios excluded by filter marked as passed.

[Andrey Tolpeev]

Any news on this?

[John Smart]

Have you tried with 0.9.262?  

[Andrey Tolpeev]

John, 

On the latest version(0.9.262) the behavior is a bit different.

Excluded by meta filter tests are still marked as Passed. Failed tests in maven build log are marked as failed, but in reports marked as Passed.

Should i provide more detail on this? How can i help to reproduce this?

[Jan De Wael]

I experienced the same: When filtering out JBehave scenario's, the scenario's are still reported as PASSED with 0 steps executed. This gives a false perception.

I found the behaviour has changed deliberatly in thucydides-core/src/main/java/net/thucydides/core/model/TestResultList.java

Tests with no steps are no longer considered pending

JUnit Tests with no steps are no longer considered PENDING unless they are marked with the @Pending annotation. This makes it easier to integrate simple tests that don't need to use the step hierarchy.

See Commit comment

While the change supports creation of JUnit based tests, it has this undesired effect for JBehave scenario's.

I would prefer to:
- support the @skip meta data in JBehave scenario's to skip execution of test, but report the scenario as PENDING.
- support the @ignore meta data in JBehave scenario's to skip both execution and reporting.
- in other cases, where scenario's are excluded due to a filter, the scenario should also not being reported.

Skip could be used to avoid a scenario to executed because it needs further attention. E.g. because it has not been completed. An alternative keyword could be @wip (work in progress, as in Cucumber)
Ignore could be used to really forget about the scenario.

[John Smart]

Hi Jan,

In JBehave, a scenario is considered passing if all of the step definitions are implemented, even if there is no code. This is because there is no obligation to use step libraries within the step definitions, though it is a good practice for more complex tests. Consider the following scenario:

Scenario: Logging on via Facebook

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

When you execute this with no step definitions, it will be reported as Pending:

When you implement the steps, they will be considered successful unless an exception is thrown or a step is marked as pending. So the following will indeed pass:

    @Given("$username has registered online via Facebook")

    public void has_registered_via_facebook(String username) {}

This is because there is no way to know that a step definition is empty - we can only know that no @Step methods were called, which does not necessarily mean that it is empty.

You can make this a pending step by using the org.jbehave.core.annotations.Pending pending annotation, e.g:

    @Pending

    @Given("$username has registered online via Facebook")

    public void has_registered_via_facebook(String username) {}

JBehave and Thucydides will now report this scenario as pending, even though it has an "implemented" (albeit empty) step definition:

This is also a good way to keep track of work if you are driving the code from the step definitions, as you can easily see which steps have been done at any point in time.

The @skip and @ignore metatags are currently synonyms, with the default behaviour being to skip these stories during the test execution. In the next release of thucydides-jbehave (should be out in the next couple of days), you will also be able to use the @pending and @wip annotations directly within the story files, e.g.

Meta:

@pending

Scenario: Logging on via Facebook

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

Scenario: Logging on via Twitter

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

or

Scenario: Logging on via Facebook

Meta:

@pending

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

--

You received this message because you are subscribed to the Google Groups "Thucydides Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jan De Wael]

Hi John,

 

I’m aware of the pending step behavior.

The issue is specific on the filtering-out scenario’s by either:

-          Marking the scenario with the Meta: @skip

-          Using a filter at command line –Dmetafilter

 

Scenario: Logging on via Facebook

Meta:

@skip

 

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

 

(observed in 0.9.262)

 

1.  Filtering at story level

 

When used at the story level, the complete story is not executed and does not show up in the report (as expected)

 

Authentication

 

As a user

I want to use my social media credentials

In order to get access to the Frequent Flyer portal

 

Meta:

@skip

 

Scenario: Logging on via Facebook

 

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

 

2.  Filtering at scenario level

 

When used at the scenario level, the scenario (gherkin) steps are not executed, but the scenario title is included in the report as PASSED but with 0 steps executed.

Scenario: Logging on via Facebook

Meta:

@skip

 

Given Joe is a Frequent Flyer member

And Joe has registered online via Facebook

When Joe logs on with a Facebook token

Then he should be given access to the site

 

Scenario: Logging on via GMail

 

Given Joe is a Frequent Flyer member

And Joe has registered online via GMail

When Joe logs on with a GMail token

Then he should be given access to the site

 

Rgds,

 

Jan

---

Geen virus gevonden in dit bericht.
Gecontroleerd door AVG - www.avg.com
Versie: 2014.0.4744 / Virusdatabase: 4007/8004 - datum van uitgifte: 08/08/14
Interne Virusdatabase is verouderd.

[John Smart]

Hi Jan,

Could you try with version 0.9.268 for both thucydides and thucydides-jbehave?

Regards

John.

[Theo]

Is there a way to have a story file showing up in the reports with 0 automated tests associated with?

I am using Thucydides + JBehave 0.9.268.

Assume I have a system capability/functionality to transfer data from/to external parties. Created a story file called data_transfer.story which contains just

Data transfer

Meta:

@ignore/@wip/@skip/@pending

If i use @ignore, nothing will appear in the reports

If i use either of @wip/@skip/@pending, the story will appear in the reports with 1 passing TC - the name of the story.

Scenario which needs to be covered: one wants to show in reports that you have in mind to cover automated acceptance testing of Data Transfer functionality, but there are currently 0 test cases for it.

If I don't create the story file or add @ignore, it will not show up in the reports and you'll always get the question "but Data Transfer is not on the list to be tested?".

If you add the story file, it will show up green in the reports with 1 passing TC, looks just strange and unnatural :D

Theo

[John Smart]

You use the Requirements view for this. Suppose you have an empty story, e.g.

Logging on to the 'My Flying High' web site using Oauth

Narrative:

In order to encourage travellers to book with Flying High Airlines more frequently

As the Flying High sales manager

I want travellers to earn Frequent Flyer points when they fly with us

Thucydides will report this as a pending requirement in the Requirements view:

and

--

You received this message because you are subscribed to the Google Groups "Thucydides Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Theo]

Makes sense now, thanks for your answer.

Theo