New Test status possible?

[Benjamin]

I'm sorry if i use the wrong terminology ;-)

TestNG offers three different Test Resultes (Failed (red),
Success(green) and Skipped (Yellow)).

Is it possible to add a fourth one: Test ended with Warnings(Orange),
that is also visible in the TestNG Eclipse Plugin?

Testcases ended with Warnings should be handeld like in the "verify"-
approach. So all other tests could continue running and it doesn't
take any influences on the test. It is just a reminder for the tester.
Would this be possible to integrate?

Thanks for your reply!

[Cédric Beust ♔]

Hi Benjamin,

Adding a status has a lot of repercussions on the whole process so the new status needs to be really important, and I'm not convinced that "had warnings" is that useful (I don't think I've ever needed it). Maybe you could use Reporter.log() to give the warnings and check the reports at the end to see if any warnings were emitted?

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[Jeff]

I second the addition of "warning" states.  I have test cases that check some web pages (Selenium) that are formatting or content, but don't affect functionality.  I don't want to pass them, but I don't want to fail them and cause test execution to stop.

We have a rudimentary way to do "soft asserts", but it would be nice to have a more clean way to mark a test result as a "warning" while still allowing execution to continue.

Maybe the test state isn't the issue, but a way to tell TestNG that if the specified @Test fails, to allow execution to continue, such as:

@Test(haltOnFailure = false)

--

Jeff Vincent
preda...@gmail.com
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

I ♥ DropBox !! 

[Benjamin]

I can totally agree with Jeff.

Actually i have a similiar situation as Jeff. We are also Testing with
Selenium some functionialities on a Web-Surface.
But sometimes the Server Console (for the Webapplikation) throws
Exceptions which dont affect the Surface. So the Testcase from the
viewpoint of Selenium works correctly and should pass, so that other
Testcases which depend on them could start.

But I want to make visible that there are Errors from the Serverside.

Surely i could Log these errors and at the end of the whole TestSuite
i could (maybe) set the Testcases which had an Server-error to "Fail",
to show this. But in my Opinion this is just a dirty hack. In my
opinion another Testcase State like "Warning" would make much sense
and it would be nice, if TestNG would offer this feature. Or at Least
to add a visible "Note" to a "Success"-Testcase?

I can imagine that this will bring much work for you Cedric ;-) , but
as the market for Surface-Testing based on Selenium is very huge and
will grow (Mobile) this could become very important, because there
will be always erros besides the Surface which are not visible, but
remarkable.

On 6 Jul., 18:07, Jeff <predato...@gmail.com> wrote:
> I second the addition of "warning" states.  I have test cases that check
> some web pages (Selenium) that are formatting or content, but don't affect
> functionality.  I don't want to pass them, but I don't want to fail them
> and cause test execution to stop.
>
> We have a rudimentary way to do "soft asserts", but it would be nice to
> have a more clean way to mark a test result as a "warning" while still
> allowing execution to continue.
>
> Maybe the test state isn't the issue, but a way to tell TestNG that if the
> specified @Test fails, to allow execution to continue, such as:
>
> @Test(haltOnFailure = false)
>
> On Fri, Jul 6, 2012 at 9:30 AM, Cédric Beust ♔ <ced...@beust.com> wrote:
>
>
>
>
>
>
>
>
>
> > Hi Benjamin,
>
> > Adding a status has a lot of repercussions on the whole process so the new
> > status needs to be really important, and I'm not convinced that "had
> > warnings" is that useful (I don't think I've ever needed it). Maybe you
> > could use Reporter.log() to give the warnings and check the reports at the
> > end to see if any warnings were emitted?
>
> > --
> > Cédric
>

> predato...@gmail.com

> See my LinkedIn profile at:http://www.linkedin.com/in/rjeffreyvincent

> I ♥ DropBox <http://db.tt/9O6LfBX> !!

[Tarun Kumar]

+1 for @Test(haltOnFailure = false)

On Friday, July 6, 2012 9:37:41 PM UTC+5:30, Jeff wrote:

I second the addition of "warning" states.  I have test cases that check some web pages (Selenium) that are formatting or content, but don't affect functionality.  I don't want to pass them, but I don't want to fail them and cause test execution to stop.

We have a rudimentary way to do "soft asserts", but it would be nice to have a more clean way to mark a test result as a "warning" while still allowing execution to continue.

Maybe the test state isn't the issue, but a way to tell TestNG that if the specified @Test fails, to allow execution to continue, such as:

@Test(haltOnFailure = false)

On Fri, Jul 6, 2012 at 9:30 AM, Cédric Beust ♔ <ced...@beust.com> wrote:

Hi Benjamin,

Adding a status has a lot of repercussions on the whole process so the new status needs to be really important, and I'm not convinced that "had warnings" is that useful (I don't think I've ever needed it). Maybe you could use Reporter.log() to give the warnings and check the reports at the end to see if any warnings were emitted?

-- 

Cédric

On Fri, Jul 6, 2012 at 8:02 AM, Benjamin <b.soel...@googlemail.com> wrote:

I'm sorry if i use the wrong terminology ;-)

TestNG offers three different Test Resultes (Failed (red),
Success(green) and Skipped (Yellow)).

Is it possible to add a fourth one: Test ended with Warnings(Orange),
that is also visible in the TestNG Eclipse Plugin?

Testcases ended with Warnings should be handeld like in the "verify"-
approach. So all other tests could continue running and it doesn't
take any influences on the test. It is just a reminder for the tester.
Would this be possible to integrate?

Thanks for your reply!

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[edwolb]

I'm going to throw a vote into Cedric's camp.  We have enough problems trying to get people to pay attention to test failures altogether.  Having warnings will be an excuse for individuals to not prioritize addressing test failures.  We implemented a "Verify" object that would allow us to capture a collection of results, and to fail against them after the fact.  This allows the test to continue, but ultimately fails it at the end.  It looks like this:

///

Verify v = new Verify();

v.verifyFalse(test1, "Testing field 1");

v.verifyTrue(test2, "Testing field 2");

v.verifyEqual(value1, value2, "Testing fields equal);

Assert.assertTrue(v.isPassed(), v.error());

///

v.isPassed() collects all results and provides an overall pass/fail.  v.error() provides a summary of which tests failed, allowing a tester to still identify where the failure occurred.  If this same functionality could occur using a flag, then perhaps that's a good answer, and we can go back to using Assert for these larger tests.  However a failure is a failure, and calling it anything else just makes it "not failed".

--

Chris

-- 

Cédric

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Benjamin]

You can always ignore Test-Failures if you want, that depends not on
"Is there a verify or not". ;-)
Before i started working in my Department, they just Comment failing
Testcases out of the Tests. So if you want, you can always take
testing not serious.

In my opinion TestNG is a Testframework that offers many possibilities
to create Tests (For example depended Tests). If the Tester wants to
use such a feature like to verify something, why should TestNG not
offer it? That should be the question.

I will now try to implement for every Testcase a dependent
"Check_Server"-Testcase, which checks if there are any errors or
warnings in the server console. So it will not influence the depended
Surface-Testing but i have an opinion to show that there are maybe
errors/warnings besides the surface.

> > On Fri, Jul 6, 2012 at 8:02 AM, Benjamin <b.soeltenf...@googlemail.com>wrote:
>
> >> I'm sorry if i use the wrong terminology ;-)
>
> >> TestNG offers three different Test Resultes (Failed (red),
> >> Success(green) and Skipped (Yellow)).
>
> >> Is it possible to add a fourth one: Test ended with Warnings(Orange),
> >> that is also visible in the TestNG Eclipse Plugin?
>
> >> Testcases ended with Warnings should be handeld like in the "verify"-
> >> approach. So all other tests could continue running and it doesn't
> >> take any influences on the test. It is just a reminder for the tester.
> >> Would this be possible to integrate?
>
> >> Thanks for your reply!
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "testng-users" group.
> >> To post to this group, send email to testng...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> testng-users...@googlegroups.com.

[Cédric Beust ♔]

On Thu, Jul 12, 2012 at 4:15 AM, Benjamin <b.soel...@googlemail.com> wrote:

Before i started working in my Department, they just Comment failing
Testcases out of the Tests.

If they are using TestNG, a better way is to put these tests in a group (e.g "broken") and exclude that group. This way you can easily keep track of what tests are being excluded.

-- 

Cédric

 

[Jason McSwain]

In my Testing Experience the need for a 4th status is unnecessary.  Consider the following.

1. a test should be focused for a particular feature
2. It is completely reasonable to have 2 tests that do a similar thing, and check different parts of the system.  like you mentioned above.
3. When tests are combined, into a single test, (i.e. multiple asserts) then you can result in a case where if the first assert fails then you don't end up running the entire test...
4. A test doesn't "kind of pass" or "kind of fail".  It either meets your expectations or it doesn't.  All software groups (product mgmt, Dev, & QA) should work together to set expectations and tests should be implemented to assert those expectations.
   

I run into this scenario all the time, and the solution is to split out your tests.  This can be done in 2 ways.

1. write two tests, one that does the actions and checks the UI, and another test that does those same actions and checks the server status or log...
2. the other option is to perform the test in a @BeforeMethod or @BeforeClass, and save the results you want to check in class level objects, then have 2 @Test methods implemented. one can validate UI data object and the other can validate server log object.

I spend most of my efforts writing system level tests with TestNG, from a QA perspective, and the use of a "warning" is just muddying the waters.  While testing can be complex, the results should be straight forward.  I like the way TestNG has implemented this:

• Pass: the test meets expectations.
• Fail: the test does not meet expectations.
• Blocked/Skipped: a precondition was not met. (assert in related test or setup has failed, and test was not run b/c env may be in a bad state)

(I'm not saying that a "warning status will never be needed", but i have yet to find a scenario in 12 years of testing where it was needed.)

Just my 2 cents, Thanks.

-Jason-

[edwolb]

If there was a Google+1 on this UI, I'd use it now :)

To unsubscribe from this group, send email to testng-users+unsubscribe@googlegroups.com.

[Rick Simpson]

I can understand the purist argument of just having pass/fail/skip, but the framework already embraces the idea of hiding failures--specifically the IRetryAnalyer and the success with % concepts.  Our group would love to have a way to add additional result states.  In our specific case, we work on a large website where the pages have parts of the UI rendered by other services we don't have control over.  We get lots of failures from the flakiness of those services. It would be nice to have the ability to look at the failure (we use Selenium), and if the failure is because of a 3rd party service, put that result in a bucket other than fail.  It would help us know the failure wasn't due to our deployment, and allows use to easily count the 3rd party failures work to fix that problem.

[Gajendra Jain]

Hi,

Could you please share the "Verify" code. so that we can use it?

Regards,

Gajenrda

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Krishnan Mahadevan]

Have you explored SoftAssert in TestNG? If not then you should. I think it has the soft assert capabilities that you are looking for [ fail a test at the end ]

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

To post to this group, send email to testng...@googlegroups.com.

Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/groups/opt_out.

--
Thanks & Regards
Krishnan Mahadevan

"All the desirable things in life are either illegal, expensive, fattening or in love with someone else!"
My Scribblings @ http://wakened-cognition.blogspot.com/