Executing subsequent tests when dependency fails

[Todd Wells]

Some of our integration tests rely on an external service during setup (a @BeforeX method) or teardown (@AfterX). Occasionally this service is flakey -- sometimes it might get re-deployed by another team in the middle of a long test run. The problem is that if one of the configuration methods fails, then any subsequent test that also has a dependency on that configuration method is skipped. We'd like it to attempt those tests anyway. Of course it is correct behavior for a given test method to be skipped if the @Before fails, we just don't want all subsequent tests also skipped.

Is there a way to do this with TestNG?

[Suk-Hyun Cho]

The issue is magnified when multiple test classes inherit the
configuration method without overriding it. When the configuration
method fails, all the subsequent tests across all the subclassing test
classes are skipped. Wrapping the configuration method around in try/
catch is a work-around, although not ideal.

[Cédric Beust ♔]

Hi Todd,

I just answered this question in a different thread, but Suk-Hyun is basically correct:  just catch the exception yourself so that TestNG will never see it.

If you think about it, it's really what you want:  there is no way TestNG can know when an exception is a real or a false failure, so it's just safer to always abort and let you, the developer, write the logic that decides when such an exception is acceptable and when it's not.

Does this make sense?

-- 

Cédric

On Tue, Apr 13, 2010 at 10:28 AM, Todd Wells <ttop...@gmail.com> wrote:

Some of our integration tests rely on an external service during setup (a @BeforeX method) or teardown (@AfterX). Occasionally this service is flakey -- sometimes it might get re-deployed by another team in the middle of a long test run. The problem is that if one of the configuration methods fails, then any subsequent test that also has a dependency on that configuration method is skipped. We'd like it to attempt those tests anyway. Of course it is correct behavior for a given test method to be skipped if the @Before fails, we just don't want all subsequent tests also skipped.

Is there a way to do this with TestNG?

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Suk-Hyun Cho]

To me, this makes more sense:

* Failure in @Before/AfterMethod skips a single test only.
* Failure in @Before/AfterClass skips tests in that class only.
* Failure in @Before/AfterSuite skips all tests in the suite.
...

Rather than

* Failure in @Before/AfterX skips skips subsequent tests in classes
that have or inherit this configuration method.

The reason that I would prefer that TestNG not eagerly skip tests is
that I believe that I should be the one to determine whether the
environment renders my testrun useless, not TestNG. At the least, I
wish there were a way to tell TestNG to not eagerly skip tests. For
example,

@BeforeMethod(eagerlySkip = false)
public void setUp() {
throw new RuntimeException();
}

@Test testFoo() {}
@Test testBar() {}
@Test testBaz() {}
where we would see

setUp failed
testFoo skipped
setUp failed
testBar skipped
setUp failed
testBaz skipped

The problem with try/catch is that a test would never fail due to,
say, SocketException. Since I would catch it and just log, a test
would fail with some other exception, say NullPointerException, which
makes debugging harder.

> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>

[Cédric Beust ♔]

On Tue, Apr 13, 2010 at 2:50 PM, Suk-Hyun Cho <choe...@gmail.com> wrote:

To me, this makes more sense:

* Failure in @Before/AfterMethod skips a single test only.
* Failure in @Before/AfterClass skips tests in that class only.
* Failure in @Before/AfterSuite skips all tests in the suite.

This is exactly how it's implemented, are you seeing something different?

 

...

Rather than

* Failure in @Before/AfterX skips skips subsequent tests in classes
that have or inherit this configuration method.

The reason that I would prefer that TestNG not eagerly skip tests is
that I believe that I should be the one to determine whether the
environment renders my testrun useless, not TestNG.

But you are the one that determines if your environment is useless:  if it is, do nothing (let the exception bubble up), if it is not, catch the exception.  This looks like a straightforward semantics to me...

-- 

Cédric

 

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Suk-Hyun Cho]

On Apr 13, 3:07 pm, Cédric Beust ♔ <cbe...@google.com> wrote:

> On Tue, Apr 13, 2010 at 2:50 PM, Suk-Hyun Cho <choey...@gmail.com> wrote:
> > To me, this makes more sense:
>
> > * Failure in @Before/AfterMethod skips a single test only.
> > * Failure in @Before/AfterClass skips tests in that class only.
> > * Failure in @Before/AfterSuite skips all tests in the suite.
>
> This is exactly how it's implemented, are you seeing something different?

That's not the behavior I am seeing.

If I have a test base and two test classes:

public class MyTestBase
{
private static boolean didFirstTestFail = false;

@BeforeTest
public void setUp()
{
if (!didFirstTestFail)
{
didFirstTestFail = true;
throw new RuntimeException(":'(");
}
}
}

public class MyFooTest extends MyTestBase
{
@Test
public void testFoo1()
{}

@Test
public void testFoo2()
{}
}

public class MyBarTest extends MyTestBase
{
@Test
public void testBar1()
{}

@Test
public void testBar2()
{}
}

And run MyFooTest and MyBarTest in a single run, The configuration
method fails for the first time, then all the tests are skipped in
both test classes.

[Cédric Beust ♔]

This is normal, you are using @BeforeTest, so if it fails, all the methods in that <test> tag will be skipped as well.

Maybe you meant to use @BeforeMethod?

-- 

Cédric

--

You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Suk-Hyun Cho]

Oops, sorry. I meant to say @BeforeMethod. The test I ran used
@BeforeMethod. I just re-ran the above repro tests with @BeforeMethod
instead of @BeforeTest, and I see the same behavior: 1 configuration
method fails, 3 configuration methods skip, and 4 tests skip.

> > testng-users...@googlegroups.com<testng-users%2Bunsubscribe@google groups.com>

[Cédric Beust ♔]

Your @BeforeMethod is in the base class, which means it applies to both subclasses.  So if it fails, all the test methods on both classes should be skipped.

Move that method to just one subclass class and you should only see test methods of that class get skipped.

This seems to be working as intended, am I missing something?

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--

Cédric

[Choey]

That does not make sense to me. I don't think that should be the intended behavior.

Let's say that we have

public abstract class MyAbstractTestBase

{

      @BeforeTest

      public void setUp()

      {

            if (getAnswer() != 42)

                  throw new RuntimeException("Wrong!");

      }

      abstract protected int getAnswer();

}

public class MyFooTest extends MyTestBase

      @Test

      public void testFoo1()

      {}

      @Test

      public void testFoo2()

      {}

      protected int getAnswer() 

      {

            return 1;

      }

}

public class MyBarTest extends MyTestBase

{

      @Test

      public void testBar1()

      {}

      @Test

      public void testBar2()

      {}

      protected int getAnswer()

      {

            return 42;

      }

}

Are you saying that if MyFooTest runs first, MyBarTest should fail, but when MyBarTest runs first, it should pass but not MyFooTest? Of course, this is a stupid test, but the point that I am trying to illustrate is that one test class may have different states than the others. Therefore, it is a false assumption that a configuration method always applies equally to all subclasses.

Overriding the configuration method in every subclass works, but do we really want to have an overridden method whose only job is to call super.setUp() for every subclass?

2010/4/13 Cédric Beust ♔ <cbe...@google.com>

[Cédric Beust ♔]

We discussed this not long ago, please take a look at the archives:

https://groups.google.com/group/testng-users/browse_frm/thread/6479a94dec6b5519

I still think that the code below works as expected:  if both your test classes extend the same class, why would you want a different behavior from the test methods that you are inheriting?

-- 

Cédric

[Todd Wells]

In an integration test we're depending on different external services with different levels of availability. So it's not that we "want" different behavior from a setup/before method, it's just that sometimes it will succeed and sometimes it won't. But it's wrong to assume that just because it didn't succeed on one execution that it won't succeed the next time. The tests are long-running, it can take hours for a complete test run, but if one setup method fails, then all subsequent tests which use the same setup method are skipped, which is not the desired behavior at all.

For example, a common occurrence in a @Before method in a base class is to create a new account in an external service, so each test executes with a new account context. Sometimes this fails due to an issue with the external system... maybe it was taken offline for a minute while a new war was deployed. But it would be wrong to assume that because it failed once it would fail again. 

Same with @After methods... maybe a bunch of created objects need to be deleted. But for some reason the cleanup fails for a particular test case. We don't want all subsequent test cases to not be executed just because one test case failed to clean up after itself as expected.

Having to try/catch in @Before and @After ends up being a crude implementation and prone to it's own problems as Choey pointed out.

Being in an inherited class is besides the point -- all of these tests could be in the same class and the @BeforeMethod could be in the same class, but we don't want to skip all remaining tests with the same @BeforeMethod just because it failed once during execution. Just report that execution as failed and carry on.

You've made your perspective pretty clear, but I think you should consider that this is actually a pretty common scenario people run in to. I was talking to an engineer at another company just today and discussing this issue and he said this is an issue for their company, too...they lots of integration tests interacting with external systems and sometimes unpredictable failures occur in a config method. But presuming that a failure in setup or teardown one time means a failure the next time is a fallacy. Yes, I would agree for unit tests this is the correct perspective, but one of the nice things about TestNG is that it doesn't usually presume that your test is a unit test.

Best regards,

-Todd

2010/4/13 Cédric Beust ♔ <cbe...@google.com>

[Cédric Beust ♔]

Hi Todd,

So how would you suggest for TestNG to support your scenario?

In the archive discussion I linked earlier, I mention an attribute called "strict", would it help with this scenario?

-- 

Cédric

[Todd Wells]

If I understood your description correctly, yes, I think that would be exactly the type of solution we're looking for. I think perhaps the "eagerlySkip" name that Choey used is a little more descriptive, but the precise name isn't that important, the functionality is :-)  

So if strict="true" on a config method (@BeforeX/@AfterX), then a failure in that config method wouldn't result in the remainder of tests being skipped? You were describing it in that thread for @BeforeClass/@BeforeTest, but I think @BeforeMethod as described by John Smith would be the most important case for us. 

-Todd

2010/4/14 Cédric Beust ♔ <cbe...@google.com>

[Cédric Beust ♔]

Actually, no, that's not what "strict" would do.  "strict" would be used in configuration methods that are inherited by many subclasses and it would allow a failure to only apply to one subclass instead of all of them.

If both A and B extend Base:

Base#before()   <--- fails

A#test   <--- skipped

Base#before()  <--- skipped

B#test   <--- skipped

with strict=true:

Base#before()   <--- fails

A#test   <--- skipped

Base#before()  <--- runs normally

B#test   <--- runs normally

--

Cédric

[Todd Wells]

Well, I think that's better than the current behavior, but it still wouldn't help with the case where you have 10 @Tests in a class that has a @BeforeMethod -- if the @BeforeMethod fails on the execution of @Test #2, then the remaining tests in that class are not executed. That's really the behavior that's problematic -- the case where @BeforeMethod lives in a base class is just a variation on this problem. 

The presumption that if @BeforeMethod fails once it will always fail is an inaccurate presumption. Same thing if the same class has an @AfterMethod -- perhaps it's deleting objects that were created on an external system. If that fails, we don't want to stop executing all the remaining tests in the class merely because the clean up of one test failed. But we don't want to swallow all errors with a try/catch in @AfterMethod in order to avoid this... we still want to know that @AfterMethod failed, we just want it to report the failure and execute the next test in the class, not skip all the remaining tests.

Do you understand our perspective here? It doesn't seem like our use case could be that uncommon for integration testing.

Thanks again for your consideration,

-Todd

2010/4/14 Cédric Beust ♔ <cbe...@google.com>

Actually, no, that's not what "strict" would do.  "strict" would be used in configuration methods that are inherited by many subclasses and it would allow a failure to only apply to one subclass instead of all of them.

[Cédric Beust ♔]

Hi Todd,

Ok, I understand what you are asking and why a try/catch is not sufficient.

It looks like an attribute such as @BeforeMethod(continueOnError = true) is what you are looking for, correct?  (a more accurate name would be something like "dontSkipOnError" but "don't" is a bit of an odd word to use as a Java identifier because of the apostrophe).

-- 

Cedric

--

Cédric

[Choey]

I would prefer "dontSkipOnError", or perhaps "neverSkipOnError", because to me, "continueOnError" implies that the test for which the configuration method was run will not be skipped but attempted to be run, which would behave as if it was wrapped around in a try/catch block.

Thank you for considering this feature!

2010/4/14 Cédric Beust ♔ <cbe...@google.com>

[Todd Wells]

skipSubsequentTestsOnFailure="false"

skipSubsequentExecutionsOnFailure="false"

or maybe skipRemaining...

both seem to more accurately (if verbosely) describe the behavior. But ultimately the behavior is the most important part, not the attribute name.

Cedric, I appreciate your responsiveness on this issue! You've built a great tool in TestNG and maintaining flexibility in it's functionality will allow it to be used in a broader set of test scenarios.

[Jek]

I'd love to have this behavior in TestNG. We've applied a patch so
that an @AfterMethod failure doesn't skip remaining tests, but have
continued to use the default @BeforeMethod behavior. In an ideal
world, these configuration methods wouldn't fail. In fact, they don't
for our unit tests. But our functional test reality is similar to
Todd's: our system depends on various external resources that, from
time to time, experience problems.

For discussion purposes below, I'll call this proposed feature
skipRemainingOnFailure (mostly because I like that name). I see
"skipRemainingOnFailure" as a more general-encompassing feature of
"strict". TestNG will always skip tests when a direct dependency
fails. That behavior is not under discussion and should not be
changed. For example, if @BeforeClass fails, all test methods in that
same instance will be skipped (whether inherited or not). And for an
@BeforeMethod failure, the current test method will be skipped.
However, with a @BeforeClass(skipRemainingOnFailure=false) failure
will not cause any test methods in other class instances to be skipped
(same as how you describe strict=true). In a similar fashion, a
failure in @BeforeMethod(skipRemainingOnFailure=false) will not cause
any test methods outside the currently (intended to be) executing one
to be skipped.

Thank you,

-John

On Apr 14, 11:46 am, Todd Wells <ttopwe...@gmail.com> wrote:
> skipSubsequentTestsOnFailure="false"
> skipSubsequentExecutionsOnFailure="false"
>
> or maybe skipRemaining...
>
> both seem to more accurately (if verbosely) describe the behavior. But
> ultimately the behavior is the most important part, not the attribute name.
>
> Cedric, I appreciate your responsiveness on this issue! You've built a great
> tool in TestNG and maintaining flexibility in it's functionality will allow
> it to be used in a broader set of test scenarios.
>

> On Wed, Apr 14, 2010 at 10:06 AM, Choey <choey...@gmail.com> wrote:
> > I would prefer "dontSkipOnError", or perhaps "neverSkipOnError", because to
> > me, "continueOnError" implies that the test for which the configuration
> > method was run will not be skipped but attempted to be run, which would
> > behave as if it was wrapped around in a try/catch block.
>
> > Thank you for considering this feature!
>
> > 2010/4/14 Cédric Beust ♔ <cbe...@google.com>
>
> >> Hi Todd,
>
> >> Ok, I understand what you are asking and why a try/catch is not
> >> sufficient.
>

> >> It looks like an attribute such as @BeforeMethod(*continueOnError *=

> >> true) is what you are looking for, correct?  (a more accurate name would be
> >> something like "dontSkipOnError" but "don't" is a bit of an odd word to use
> >> as a Java identifier because of the apostrophe).
>
> >> --
> >> Cedric
>

> >>>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >>>> .
> >>>> For more options, visit this group at
> >>>>http://groups.google.com/group/testng-users?hl=en.
>
> >>>  --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "testng-users" group.
> >>> To post to this group, send email to testng...@googlegroups.com.
> >>> To unsubscribe from this group, send email to

> >>> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/testng-users?hl=en.
>
> >> --
> >> Cédric
>
> >>  --
> >> You received this message because you are subscribed to the Google Groups
> >> "testng-users" group.
> >> To post to this group, send email to testng...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/testng-users?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

[Jek]

Hi Cedric,

I'm curious if any work has been done on this? We're willing to take
on the development of a skipRemainingOnFailure=boolean attribute for
the @BeforeMethod, @AfterMethod, @BeforeClass, and @AfterClass
annotations with the latter 2 behaving as the previously spec'ed
strict=boolean. Would you accept a patch of this work into the TestNG
codebase? (provided you approve of how it was implemented of course!)

Thanks,

-John

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

Hi Jek, Todd and Choey,

I actually took a look at what it would take to implement this and it turns out not be too hard, so there is no real technical objection to adding this to TestNG (I actually have a working implementation on my machine).

Rereading the thread, I still have doubts, though, and I'm no longer sure I understand why it's needed.

My fundamental objection remains that if you want a configuration failure to be ignored, you should try/catch it.  It lets you, the user, target exactly what exceptions you are willing to ignore and which ones should still cause a failure, as opposed to an attribute which is basically a boolean (ignore all failures or none).

When I mentioned the try/catch approach, Todd pointed me to a previous answer:

Having to try/catch in @Before and @After ends up being a crude implementation and prone to it's own problems as Choey pointed out.

So I went back to Choey's message and I'm still not seeing a compelling argument against try/catch.  The main disagreement I see is this:

Are you saying that if MyFooTest runs first, MyBarTest should fail, but when MyBarTest runs first, it should pass but not MyFooTest?

Yes, that's exactly what I'm saying.  Regardless of the ordering of the tests, once a @BeforeMethod fails, something wrong (that even you, the user, didn't anticipate) happened, and continuing to run the tests from then on will lead to garbage results.  If a few test methods ran successfully before the failure happened, then these results are correct since the configurations executed properly.

Choey's example can be very easily fixed by adding a try/catch in the @BeforeTest method of the super class around the call that can possibly throw.  What is so different between:

@BeforeTest(dontSkipOnError = true)

public void setUp() {

  init();   // can sometimes throw

}

and

@BeforeTest

public void setUp() {

  try {

    init();   // can sometimes throw

  } catch(MyException ex) {

    // ignore

  }

}

?

This latter example seems to be much more correct since it will only ignore exceptions that you know are safe to ignore, but your test will still bomb if something unexpected happens.  This seems like a much better solution.

Todd writes:

For example, a common occurrence in a @Before method in a base class is to create a new account in an external service, so each test executes with a new account context. Sometimes this fails due to an issue with the external system... maybe it was taken offline for a minute while a new war was deployed. But it would be wrong to assume that because it failed once it would fail again.

If you know that account creation can fail and that it's no big deal when it does, just catch AccountCreationException and just do nothing, so same question to you:  why is an attribute "dontSkipOnError" a better solution?

-- 

Cedric

[Choey]

The request for this feature comes from the assumption that in the absence of test dependencies, one test is independent of the result of another test. Extending this notion, one test should be independent of the result of another test's setup phase.

Consider the account creation example. If an account creation fails, say, because the dependent service(s) went down, we want to fault and let the test fail. However, we want to give subsequent tests a chance at running, in case the dependent service(s) come online, again. Try/catch in the @BeforeMethod works, but not quite--when the account creation fails, it will ignore that and proceed with the test method, which will fail for some odd reason (e.g., NPE because the account variable is not initialized).

2010/4/21 Cédric Beust ♔ <cbe...@google.com>

[Todd Wells]

Because again, if @BeforeMethod is interacting with an external system, we will probably not be able to anticipate what the precise failure is... maybe it's just a timeout. We still want the failure to be reported (not just buried in a log statement logged by the catch), but to presume that because @BeforeMethod failed the first time it will fail in subsequent executions is a bad presumption.

So you are presuming we know precisely what will fail (we don't, it will probably be a runtime exception) and that catching it is "good enough" -- but then we essentially lose the information that it failed (without having some separate process that greps logs) and the subsequent dependent test method will almost certainly fail in some unexpected way, so we'd rather not execute the dependent test if the @BeforeMethod failed. 

A dontSkipOnError=true attribute (not my favorite name) would allow the error to be reported correctly, the dependent test method to be skipped as expected, but would not skip all subsequent tests that depend on the same @BeforeMethod. This would be a much preferred solution to me, and it sounds like others see that as the preferred solution, too. Since it's not default behavior, it doesn't hurt anybody who doesn't explicitly enable it.

2010/4/21 Cédric Beust ♔ <cbe...@google.com>

Yes, that's exactly what I'm saying.  Regardless of the ordering of the tests, once a @BeforeMethod fails, something wrong (that even you, the user, didn't anticipate) happened, and continuing to run the tests from then on will lead to garbage results.  If a few test methods ran successfully before the failure happened, then these results are correct since the configurations executed properly.

Choey's example can be very easily fixed by adding a try/catch in the @BeforeTest method of the super class around the call that can possibly throw.  What is so different between:

@BeforeTest(dontSkipOnError = true)

public void setUp() {

  init();   // can sometimes throw

}

and

@BeforeTest

public void setUp() {

  try {

    init();   // can sometimes throw

  } catch(MyException ex) {

    // ignore

  }

}

?

This latter example seems to be much more correct since it will only ignore exceptions that you know are safe to ignore, but your test will still bomb if something unexpected happens.  This seems like a much better solution.

Todd writes:

For example, a common occurrence in a @Before method in a base class is to create a new account in an external service, so each test executes with a new account context. Sometimes this fails due to an issue with the external system... maybe it was taken offline for a minute while a new war was deployed. But it would be wrong to assume that because it failed once it would fail again.

If you know that account creation can fail and that it's no big deal when it does, just catch AccountCreationException and just do nothing, so same question to you:  why is an attribute "dontSkipOnError" a better solution?

--

[Cédric Beust ♔]

Okay I realize that I had misunderstood (again) exactly what you are asking.

I thought you wanted to ignore all the configuration failures, but what you want is for the failure of that particular configuration method to be reported (and to cause skips as required) but not to cause skips later.

Now, specifying this semantics is tricky because it will vary for each configuration method.

For example, for @BeforeMethod, with that flag set:

bm()  <--- fails

a()    <--- skip or run it normally?

bm()  <--- run normally, might succeed or fail

b()    <--- depends

Now, what does it mean for a @BeforeClass?  The tricky case is when that method is in a base class, and therefore reused in different classes:

A#bc()   <--- fails

A#a()   |

A#b()   | --- skip or run?

A#c()   |

B#bc()   <--- run normally, might succeed or fail

B#d  |__ run normally

B#e  |

There are three more configuration methods to go over...

-- 

Cedric

--
Cédric

[Todd Wells]

2010/4/21 Cédric Beust ♔ <ced...@beust.com>

I thought you wanted to ignore all the configuration failures, but what you want is for the failure of that particular configuration method to be reported (and to cause skips as required) but not to cause skips later.

Yes! Exactly... not to cause skips later.

 

Now, specifying this semantics is tricky because it will vary for each configuration method.

For example, for @BeforeMethod, with that flag set:

bm()  <--- fails

a()    <--- skip or run it normally?     skip

bm()  <--- run normally, might succeed or fail

b()    <--- depends skip if second bm() failed, otherwise run 

 

Now, what does it mean for a @BeforeClass?  The tricky case is when that method is in a base class, and therefore reused in different classes:

We don't use @BeforeClass (which doesn't seem to work well with Groovy, which we use for coding tests), but the behavior I would expect is: 

A#bc()   <--- fails

A#a()   |

A#b()   | --- skip or run? skip all three 

A#c()   |

B#bc()   <--- run normally, might succeed or fail

B#d  |__ run normally   skip all if B#bc failed, otherwise execute normally

B#e  |

There are three more configuration methods to go over...

I think you can probably extrapolate the expected behavior for those based on my answers above.

My preferred nomenclature would be something like skipSubsequentExecutionsOnFailure="false" or eagerlySkip="false", where the current behavior would be considered "true".

[Cédric Beust ♔]

Ok.

And yes, based on my new understanding, "dontSkipOnError" is not the right name.  "skipSubsequentExecutionsOnFailure" is the most accurate bit a bit of a mouthful.  Maybe "keepRunningOnFailure"...

-- 

Cedric

--
Cédric

[Jek]

Hello,

Jumping in here with a some naming ideas:

keepRunningOnFailure - not as good, it implies that the currently
executing test method will run.

runSubsequentTestsOnFailure

skipSubsequentTestsOnFailure

runRemainingTestsOnFailure

skipRemainingTestsOnFailure

proceedWithNextTestOnFailure

continueWithNextTestOnFailure

executeNextTestOnFailure

executeNextOnFailure


-John

On Apr 21, 2:45 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Ok.
>
> And yes, based on my new understanding, "dontSkipOnError" is not the right
> name.  "skipSubsequentExecutionsOnFailure" is the most accurate bit a bit of
> a mouthful.  Maybe "keepRunningOnFailure"...
>
> --
> Cedric
>
>
>
> On Wed, Apr 21, 2010 at 2:42 PM, Todd Wells <ttopwe...@gmail.com> wrote:
> > 2010/4/21 Cédric Beust ♔ <ced...@beust.com>
>
> >> I thought you wanted to ignore all the configuration failures, but what
> >> you want is for the failure of that particular configuration method to be
> >> reported (and to cause skips as required) but not to cause skips later.
>
> > Yes! Exactly... not to cause skips later.
>
> >> Now, specifying this semantics is tricky because it will vary for each
> >> configuration method.
>
> >> For example, for @BeforeMethod, with that flag set:
>
> >> bm()  <--- fails

> >> a()    <--- skip or run it normally?     *skip*

> >> bm()  <--- run normally, might succeed or fail

> >> b()    <--- depends *skip if second bm() failed, otherwise run*

>
> >> Now, what does it mean for a @BeforeClass?  The tricky case is when that
> >> method is in a base class, and therefore reused in different classes:
>
> > We don't use @BeforeClass (which doesn't seem to work well with Groovy,
> > which we use for coding tests), but the behavior I would expect is:
>
> >> A#bc()   <--- fails
> >> A#a()   |

> >> A#b()   | --- skip or run? *skip all three *

> >> A#c()   |
>
> >> B#bc()   <--- run normally, might succeed or fail

> >> B#d  |__ run normally   *skip all if B#bc failed, otherwise execute
> >> normally*

> >> B#e  |
>
> >> There are three more configuration methods to go over...
>
> > I think you can probably extrapolate the expected behavior for those based
> > on my answers above.
>

> > My preferred nomenclature would be something like *
> > skipSubsequentExecutionsOnFailure="false"* or *eagerlySkip="false"*, where

> > the current behavior would be considered "true".
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

[Cédric Beust ♔]

Actually, I think I came up with the ultimate best name of all times:

alwaysRun.

And as it turns out, that attribute already exists!

Okay, its existing meaning conflicts somewhat with what we're trying to achieve here, but fundamentally, this is really what we are looking for:  a way to tell TestNG to always run a method, regardless of previous events.

The current meaning of alwaysRun is connected to a static decision:  what groups the user wants to run.  This attribute tells TestNG to run this method even if it doesn't belong to any of these groups.  We want to extend this attribute to factor in a runtime factor:  even if this method should be skipped because of a previous failure, run it anyway.

I think the logic is sound but I'm afraid that adding this meaning to alwaysRun might break existing code, so it's probably not a good idea.

Following this reasoning, maybe "neverSkip" is a good candidate, although I'm not a fan of negative names.

-- 

Cedric

[Jek]

Hello,

Indeed, alwaysRun works better than the rest of the earlier choices.
However, I'm starting to think we may be on the wrong track. When I
have trouble naming something it is usually an indicator of a design
issue. AFAIK all of the attributes of test configuration (@BeforeXXX
@AfterXXX) annotations refer to how each method should be looked at by
the system, primarily to see if it should be executed or not. But in
this case, we are talking about whether a configuration (and the
dependent test) should be executed in the future. This is in the
system realm. I propose we introduce a new interface,
IConfigurationMethodFailureHandler, which is called by TestNG to find
out if we should continue running other dependent tests. The default
implementation can be how TestNG behaves now. Another option is to
introduce a system-wide setting to change the behavior, but that is
not as flexible.

-John

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

[Todd Wells]

What would the method signature(s) of this interface look like? How would you tell TestNG to use that implementation?

[Jim]

I just wanted to give another scenario that we have that I think would
need a solution like this. We currently have several test classes
that extend a common base class. We're using mockito to mock out
objects representing external systems. In our base class we have a
BeforeMethod to initialize the mockito objects and an AfterMethod to
ensure that there are no interactions with the mock objects that
haven't been accounted for. We don't want to have to call this method
at the end of each test - AfterMethod works really well for this.
Unfortunately, if the first test had interactions with a mock object,
we only find out when it's checked in the AfterMethod. Clearly, this
test should fail - we don't want to catch the exception and show the
test as passing (if there's a way to have the AfterMethod cause the
failure to show up in the actual test method, this wouldn't be an
issue - please let me know if there's a way to do this). However, if
the AfterMethod fails, it shouldn't cause the subsequent tests to be
skipped since the BeforeMethod will reset everything. I also wouldn't
want to have to add @Test(alwaysRun = true) to every test method just
to support this - it would be very tedious and defeat the nice feature
of having @Test at the class level. I hope this post makes sense.

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

[Todd Wells]

I'd like to keep the ball rolling here, it seems like this issue has stalled and there are clearly a number of folks who could use this functionality. Jek, can you elaborate on the interface idea you proposed?

[Jek]

Hello,

I've run the registered handler interface idea across a few heavy
TestNG users here and the general thinking is that it's not needed.
For us, we'd like TestNG to always behave in the way described in this
message thread. Assuming that other supporters feel the same way, I
propose that a command line parameter be added to the TestNG runner
called -failurepolicy. It takes one of two possible arguments:
continue|skip. This parameter will also appear as an attribute on the
suite element in testng.xml and settable via the API. The default
failure policy will be "skip" in order to keep the behavior the same
for existing users. A failure policy of "continue" will enact the new
behavior.

We can make this change to TestNG; however, it is best for us if we
can get it into the TestNG codebase.

-John

P.S. Sorry for the late response on this.

> > > > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
> > <testng-users%252Buns...@googlegroups.com<testng-users%25252Bun...@googlegroups.com>

>
> > > > > > .
> > > > > > For more options, visit this group at
> > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > > --
> > > > > Cédric
>
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "testng-users" group.
> > > > > To post to this group, send email to testng...@googlegroups.com.
> > > > > To unsubscribe from this group, send email to
> > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>
> > > > .

[Nalin Makar]

+1 for this suggestion. Having this functionality in TestNG is a good idea.

-nalin

[Todd Wells]

I think that name is the best one I've heard so far. If it is going to be a command-line option, it will also need to be exposed to the testng ant task. My other concern there is how IDEs could be adapted to use this behavior.

[Jek]

Hi Todd,

I agree on the Ant task. As for IDEs, we use IDEA. In IDEA, you can
set "Test runner parameters" in the default TestNG run configuration.
The default configuration is used as a template for auto-generated run
configurations created when running a test class or method. I'm sure
there is something similar in Eclipse.

-John

On May 20, 6:12 am, Todd Wells <ttopwe...@gmail.com> wrote:
> I think that name is the best one I've heard so far. If it is going to be a
> command-line option, it will also need to be exposed to the testng ant task.
> My other concern there is how IDEs could be adapted to use this behavior.
>

> On Wed, May 19, 2010 at 10:22 PM, Nalin Makar <nalin.ma...@gmail.com> wrote:
> > +1 for this suggestion. Having this functionality in TestNG is a good idea.
>
> > -nalin
>

> >> > > > > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
> >> <testng-users%252Buns...@googlegroups.com<testng-users%25252Bun...@googlegroups.com>
>
> >> > > <testng-users%252Buns...@googlegroups.com<testng-users%25252Bun...@googlegroups.com>
> >> <testng-users%25252Bun...@googlegroups.com<testng-users%2525252Bu...@googlegroups.com>

>
> >> > > > > > > .
> >> > > > > > > For more options, visit this group at
> >> > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> >> > > > > > --
> >> > > > > > Cédric
>
> >> > > > > > --
> >> > > > > > You received this message because you are subscribed to the
> >> Google
> >> > > Groups
> >> > > > > "testng-users" group.
> >> > > > > > To post to this group, send email to
> >> testng...@googlegroups.com.
> >> > > > > > To unsubscribe from this group, send email to
> >> > > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> >> <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>

> >> > > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>
> ...
>
> read more »

[Todd Wells]

Cedric, if John's team implements this, will you accept a patch? Does the proposal meet with your approval?

[Cédric Beust ♔]

On Tue, May 25, 2010 at 12:11 PM, Todd Wells <ttop...@gmail.com> wrote:

Cedric, if John's team implements this, will you accept a patch? Does the proposal meet with your approval?

Yes, although I reserve the right to change my mind once I see the patch :-)

--
Cédric

 

--
Cédric

[toddq]

Here's the patch that implements this - http://www.quessenberry.com/configFailurePolicy_922.diff.
Let me know what you think.

ToddQ

On May 25, 12:41 pm, Cédric Beust ♔ <ced...@beust.com> wrote:

> ...
>
> read more »

[Jek]

We've been testing the patch and have found a couple of issues. We're
working directly with ToddQ. He expects there will be an updated
patch out very soon. The good news is that it is very close. Thanks
Todd!

-John

On Jun 15, 11:23 am, toddq <todd.quessenbe...@gmail.com> wrote:
> Here's the patch that implements this -http://www.quessenberry.com/configFailurePolicy_922.diff.

> ...
>
> read more »

[Cédric Beust ♔]

Ok, I'll wait for the updated version.

--
Cédric

> ...
>
> read more »

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[toddq]

Here's an updated patch - http://www.quessenberry.com/configFailurePolicy_924.diff.
It's passed review and testing by John (Jek).

ToddQ

> ...
>
> read more »

[Jek]

It indeed looks good. Thanks ToddQ!

A quick review on how to control this setting:
Command line: -configfailurepolicy continue|skip
testng.xml: <suite configfailurepolicy="continue|skip" ...>
Ant task: <testng configfailurepolicy="continue|skip" ...>

Feature recap: When configfailurepolicy is set to 'continue', TestNG
will continue to run any remaining tests that do not depend on the
"instance" of a configuration method that fails or encounters a
SkipException. For example, if a @BeforeClass method in a base class
with 2 subclasses fails when running for subclass 1, subclass 2 will
not be automatically skipped. (Subclass 1 will be skipped as
expected.) In a similar way, if a local @BeforeMethod method fails
for test 1, test 2 will not be automatically skipped. (Test 1 will
though.)

Note: the default behavior is "skip" and is the same TestNG behavior
as previous.

The main driver for this feature is handling config methods which have
nondeterministic behavior (usually external dependencies).

-John

On Jun 18, 2:06 pm, toddq <todd.quessenbe...@gmail.com> wrote:
> Here's an updated patch -http://www.quessenberry.com/configFailurePolicy_924.diff.

> ...
>
> read more »

[Choey]

Cool.

Typo on "continuel"?

[Choey]

Oh, never mind. The pipe looked like an l on my screen.

On Jun 18, 2010, at 5:17 PM, Jek <john.m...@attachmate.com> wrote:

[Nalin Makar]

>> Here's an updated patch - http://www.quessenberry.com/configFailurePolicy_924.diff.
>> It's passed review and testing by John (Jek).

Cedric, did you get a change to play with this patch?

-nalin

[Ivan ✪]

So, are you still considering including this feature?

Ivan.-

[Cédric Beust ♔]

-configfailurepolicy has been available in 5.13.1 for about a month now.

-- 

Cédric

2010/9/22 Ivan ✪ <ivansol...@gmail.com>

--
Cédric