skip on failure?

[Dan Fabulich]

I like TimeBombSkipException, but I don't necessarily want to *skip* a
test that I believe is buggy; I may want to run it first, but mark it
yellow if it fails. This is for two reasons:

1) The test could fail for a different reason (different bug); I'd like to
be able to skim the stacktrace in the report to make sure it looks like
it's the same bug

2) The bug could get fixed, at which point I'd like it to automatically
pass instead of being skipped until the expiration date.

Right now I can do what I want by writing boilerplate like this:

private void realFooTest() { /* ... */ }
@Test public void fooTest() {
try {
realFooTest();
} catch (Throwable t) {
if (t instanceof ThreadDeath) throw t;
TimeBombSkipException tbse = new TimeBombSkipException("Bug 39518", "2008/03/21");
tbse.initCause(t);
throw tbse;
}
}

... but that's actually quite a bit of boilerplate. It'd be nice if I
could just do this:

@Test(skipOnFail=true) public void fooTest() { /* ...*/ }

or, even better:

@Test
@SkipOnFail(until="2008/03/21", reason="Bug 39518")
public void fooTest() { /* ... */ }

Does this sound useful?

-Dan

[Benjamin Damm]

Actually, the idea of a soft-fail as being a legitimate result (along
with success, faailure, and skip) has come up in my work as well. We
would like to flag a test when we have acknowledged it is broken and
filed an issue for it, perhaps like this:

@Test
@Issue(id="bug-id")

public void fooTest() { /* ... */ }

A "skip-on-fail" would solve this too. Tests that have an issue ID
should be run, but don't need to be sounding alarm bells when they
fail.

-Ben

[Benjamin Damm]

What about using the "onTestFailedButWithinSuccessPercentage" result
for this kind of test condition? It's reported differently from
straight out "success".

On Mar 9, 10:38 pm, Benjamin Damm <slightly.less.ran...@gmail.com>
wrote:

[Mark Derricutt]

This @Issue idea is something I implemented last year (@RelatedIssue) - currently it only works with JIRA, you can find the code at:

http://code.google.com/p/testng-issue-reporter/

I should really update/add some wiki pages to the project as well.  Currently, the way the add-on works is that it will raise a jira ticket referencing which tests failed, with which stacktraces, and which related issues may be affected.

Some related blog posts on it:

http://www.talios.com/automated_test_regression_reporting_with_testng_and_jira.htm
http://www.talios.com/combining_test_regression_failure_reports.htm